扇贝单词前端 JavaScript 逆向

在写扇贝单词的 Chrome 插件中,需要使用一些非公开的 API,在此记录一下遇到的波折。

插件 Github 链接:https://github.com/zerolfx/shanbay-helper

由于需要完成任意网页查词的功能(扇贝只支持自己网页内的查词),所以要先观察一下,在扇贝网页内查词时调用了什么 API。借助 Chrome 的开发者工具中的 Network,很容易定位到这个请求。这是一个 GET 请求,请求的 URL 为 https://apiv3.shanbay.com/wordsapp/words/vocab?word=reverse(以单词 reverse 为例),返回的 JSON 信息如下。

1
{"data": "T5SXR44XQRUOCNXHCXNQYAP22AOCMSOCCFTGYAPMNBYAP256CO4XOCNXHCXNQ4XOCCFOCORIH2AGEORXYATSTGX2AGEOPP2TT7IEZXHZ2PTGR44XQRUIHK363P2CFURYA2EYABYAIHORCPVZURJX56G32LP256CFWUVTSURHZVMHZCFWUVTSUREWZOTSIH2AURYAPMNBYAGJCFA7QCFTWVMG2XHZ2PTGX2AQIHP2SCB4XGEEWL2AP2ORCPVZURJXVZG3VZCPVZURJQEZURWURYAPMNBYAG3CFVMEZGJTTCO63P2CFGE63PEEW3OKOC2LXHCXNQEWURKZO252EJL25URHZIHTS2EKZORMOCEWXTWURRMAEWUR56VMYA2ESCG3MSOCHZZO4XURSCGEEWGENXSNXNQ7QP2TT7TRUPEVZCOVZGJVZCOU3G3TTVZORXHZ2PTGXHZOCHZGEHZZORMGEEWXRMGETT3EWOCHZXTWURJQOR2EJXTSURRMG3CPGEHZQYAGEKUVRM2EJ3ORXYATSTGXEW3VZOCKOROPCFTWVMRMCFTWVM4XPEWURYA2EYAAPEXEW7Q25P2WARM2EYA4VOPPETTVMEZGJTTLOKG3CFVMEZGJTT4VCPPMEWQ7QR4TT7TU3PMEWGEOPPE2LCO7QG3EWGE63IHWVMYACF563VZOCKOROPPMRMLMSGERMBTSURHZTGTWOCEWVMHZUR56QHZ2ESCZOTSURKUV7IGETTUR56URHZ7QEWG3RMBTSURRMZOTSPMHZ7TYAOCSCL56OCKXTWG3RMQHZGEHZXVZGERMGEEWGERM37QGEJZORMGESC7TORGEJUV56URRMUVCPPMEWVZTSUR4XXSNXNQ2PP2W7TTSIHRM2POPPM56VZOROCKOR7QPMTT3VZOCKOROPUR2L7IYAG3CFOREZPEYA7IHZPE5625OPG3TTQHZCFTWAVZG3ORCO7QP2TT7TRUPE4X4VCP2EJIHTSURJXMSGE56OCORG3RMGEYAG3RMEZ25URJAOR2ESCVMHZGEHZXMSOCEWURTSURJUR25URNMS25G3EWUVCPGEEW7TYAGE56UR4XG3RMG34XGESCIHRMOCHZIHCPG3SCZO25URRMUV25OCSCZOVZGEHZURVZUR2L7IOKIHJURYACF2LTSTGXEW3VZOCKOROPCFTWVMG2CFTWVM4XPEWURYA2EYAAPEXEW7Q25P2WARM2EYA4VOPPETTVMEZGJTTLOKG3CFVMEZGJTT4VCPPMEWQ7QR4TT7TU3PMEWGEOPPE2LCO7QG3EWGE63IHWVMYACF563VZOCKOROPPMRMUVMSURRMQYAG3RMX56URHZVMEWURJZOCPOCHZL25GEHZ37QGEKUR56URTTX4XOCSCZOCPG3EWVMEWPMEW7TORUREWVMEWOCEWG356G3SC7TYAGETT3YAGEHZXMSOCTTGEYAG3RMG3VZURJL4XURHZIHRMG3SCTGMSPMEWVZTSUR4XXSNXNQ2PP2W7TTSIHRM2POPPM56VZOROCKOR7QPMTT3VZOCKOROPUR2L7IYAG3CFOREZPEYA7IHZPE5625OPG3TTQHZCFTWAVZG3ORCO7QP2TT7TRUPE4X4VVZ2EJUR4XG3EWUR4XGEHZXVZOCHZB25URTTG3CPGEJG3CPG3SC7THZGEHZ3YAUREWUV25URTTQOROCYA7IEZOCSCQOROCEWOCEWGEEWL25G3HZVM7QG3HZG34X2EKVMHZG3EWUR56GESCBCPURHZXTWUR56LMS2EN7IOKIHJURYACFCF25SNXNQHZIHEWVM7QP2KVMEZCF56325XHZ2PTGXHZXTSURSCTGOKURJLOKURSCAUVURSCZOHCURJZOHCGESCURU32EJLMSURRMIH56IZRMBTS2EHZBTSXYATSTGX2AVMTSOCK325OCTT7T63G3CFZOYA2EYABYAURHZBCP2E2L25TSUR4X25CPGE3ZOTS2EJ2PRM2ESC2PRM2ENMSRMGEJEZ56URRMEZG2URJBHCURJBYAPMNBYAIHEWVMEWCF56OREZXHZ2PTGXEW37QG3TT37QXYATSTGX2AOCOPG3563YACFTW7T7IIHKUVYA2EYABTSPMNBYAG356COOKPETTVMU3P2NXHCXNXYA63ZO=="}

看到大小为 32 的字符集以及最后的两个等号,就知道这是 Base32 编码了。然后用这个在线工具 https://cryptii.com/pipes/base32 (这个网站支持许多能自定义的编码方式的加解密)尝试解码,然而事情并没有那么简单,转换出来的二进制并不能显示为文本。于是随手试了试把 Base32 中的 A-Z2-7 交换前后顺序以及内部颠倒,但还是不行。所以这肯定是定制的 Base32 的,是否单单修改了编码表也不能确认。

由于解码是在前端完成的,所以我决定不如定位到对应的 JS 代码来做解密。在开发者工具的 Source 页面,可以看到扇贝的大量前端源码(React + Typescript),而且是没有经过打包前的结果。在 assets.baydn.com/web_static/words_wordsweb/static/js/services/words.ts 中找到了所有需要的 API(这个文件的源码我会贴在最后),也在其中找到了一个叫做 bayDecode 的解码函数。然后在开发者工具的 Source 页面选择在所有文件中搜索这个函数名,定位到了这个函数的定义,代码如下。

1
2
3
export const bayDecode = (str: string) => {
return JSON.parse(window.bays4.d(str));
};

可以看到有一个名叫 bays4 的全局函数做了解码的主要工作。(插一句题外话,这个函数名取得真是不错,取了 shanbay 的一部分,和 base 这个单词读音近似。)如果继续在源码中搜索 bays4,会发现除了之前那个函数的调用,只出现在了 HTML 中的 script 标签内。打开 Console,输入 bays.d,可以看到这个函数的定义,单击后能跳转到源码,果然就是嵌在 HTML 中的那个函数。

可惜的是,这个函数经过了混淆,可读性极差。但是我只需要能完成解码工作的函数,并不需要理解解码的具体过程,于是把那一大段 JS 复制到本地,用之前 API 请求来的加密信息测试了一下,果然可以使用了。最后我把那段代码嵌入到了插件的 JS 文件中,供请求 API 的时候调用。

补充:在 Source 中查看 JS 源码还是一件很痛苦的事情,但是想保存到本地的话,一次只能保存单个文件。但是在这个插件的帮助下,可以批量保存到本地,用 IDE 打开,无论是搜索还是跳转都方便了许多。

解码后结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
{
"id": "bvvxex",
"word": "reverse",
"sound": {
"ipa_us": "r\u026a'v\u025c\u02d0rs",
"ipa_uk": "r\u026a'v\u025c\u02d0s",
"audio_us_name": "f0d48142708d3fb730f3eb968f4213f4",
"audio_uk_name": "6c6436b35aff2702e82036162b4e381e",
"audio_us_urls": [
"https://media-audio1.baydn.com/abc_pub_audio/18700287fec3bc9400e95c628fc00340.4be16db7c2c62573f71a44354e456351.mp3",
"https://media-audio1.baydn.com/abc_pub_audio/18700287fec3bc9400e95c628fc00340.4be16db7c2c62573f71a44354e456351.mp3"
],
"audio_uk_urls": [
"https://media-audio1.baydn.com/abc_pub_audio/5832bc2625f041f1461a4c61b2e41bef.de2efff6a4b5ab628ecbc65012273a88.mp3",
"https://media-audio1.baydn.com/abc_pub_audio/5832bc2625f041f1461a4c61b2e41bef.de2efff6a4b5ab628ecbc65012273a88.mp3"
]
},
"created_at": "2018-01-10T14:04:53.818376+00:00",
"updated_at": "2019-03-14T08:39:38.349639+00:00",
"ref_id": "aaaaa",
"vocab_type": 0,
"comment": ""
}

代码 assets.baydn.com/web_static/words_wordsweb/static/js/services/words.ts

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
import { v3 } from 'helpers/withPrefix';
import _ from 'lodash';
import { Observable, from, defer } from 'rxjs';
import { retryWhen, concatMap } from 'rxjs/operators';
import axios, { AxiosResponse } from 'axios';
import { genericRetryStrategy } from 'helpers/rx';
import { bayDecode } from 'helpers';
import * as services from '.';

const _decodeAndKeyBy = (key: string) => (res: AxiosResponse<any>) => {
return _.keyBy(_.get(bayDecode(res.data.data), 'objects', []), key);
};

const getVocabSenses = (vocabIds: string[], dictId?: string) => {
return axios({
url: v3('/wordsapp/words/vocab_senses'),
params: {
vocab_ids: vocabIds.join(','),
dict_id: dictId,
},
}).then(_decodeAndKeyBy('vocabulary_id'));
};

const getVocabExamples = (vocabIds: string[], dictId?: string) => {
return axios({
url: v3('/wordsapp/words/vocab_examples'),
params: {
vocab_ids: vocabIds.join(','),
dict_id: dictId,
},
}).then(_decodeAndKeyBy('vocab_id'));
};

const getExtExamples = async (vocabIds: string[], dictId?: string) => {
return await axios({
url: v3('/wordsapp/words/ext_examples'),
params: {
vocab_ids: vocabIds.join(','),
dict_id: dictId,
},
}).then(_decodeAndKeyBy('vocab_id'));
};

const _genDetailsPromisesByChunkIds = (
options: GenVocabDetailStreamOptions,
chunkIds: string[],
) => {
const emptyPromise = Promise.resolve(undefined);
const promises = [
getVocabSenses(chunkIds, options.dictId),
getVocabExamples(chunkIds, options.dictId),
options.loadExtExamples ? getExtExamples(chunkIds, options.dictId) : emptyPromise,
options.loadAffixes
? services.appletsService.getAffixesByVocabIds(chunkIds)
: emptyPromise,
options.loadCollins
? services.appletsService.getCollinsSensesByVocabIds(chunkIds)
: emptyPromise,
options.loadCollins
? services.appletsService.getCollinsExamplesByVocabIds(chunkIds)
: emptyPromise,
options.loadRoots
? services.appletsService.getRootsByVocabIds(chunkIds)
: emptyPromise,
];
return promises;
};

const getVocabsDetailByIds = async (
options: GenVocabDetailStreamOptions,
chunkIds: string[],
): Promise<Array<IVocabDetail>> => {
const [
senses,
examples,
extExamples,
affixes,
collinsSenses,
collinsExamples,
roots,
] = await Promise.all(_genDetailsPromisesByChunkIds(options, chunkIds));

const details = chunkIds.map(vid => {
return {
id: vid,
sense: _.get(senses, vid),
examp: _.get(examples, [vid, 'examples'], []),
extExample: _.get(extExamples, [vid, 'examples'], []),
affixe: _.get(affixes, vid),
collinsSense: _.get(collinsSenses, [vid, 'definitions']),
collinsExample: _.get(collinsExamples, vid, []),
roots: _.get(roots, vid),
notes: undefined,
collinsVocabType: _.get(collinsSenses, [vid, 'vocab_type']),
};
});

return details;
};

const _genOneGroupVocabDetailWithRetryStream = (
options: GenVocabDetailStreamOptions,
chunkIds: string[],
): Observable<any> =>
defer(() => getVocabsDetailByIds(options, chunkIds)).pipe(
retryWhen(
genericRetryStrategy({
retryMax: 3,
delayTime: 1000,
}),
),
);

interface GenVocabDetailStreamOptions {
ids: string[];
dictId?: string;
loadExtExamples: boolean;
loadAffixes: boolean;
loadRoots: boolean;
loadCollins: boolean;
}

const genVocabDetailStream = (options: GenVocabDetailStreamOptions): Observable<any> =>
from(_.chunk(options.ids, 10)).pipe(
concatMap(chunkIds => _genOneGroupVocabDetailWithRetryStream(options, chunkIds)),
);

function fetchVocabIdsInfo$(
idList: string[],
func: (ids: string[]) => any,
): Observable<any> {
const chunks = _.chunk(idList, 10);
const reuqest = (chunkIds: string[]) =>
defer(() => func(chunkIds)).pipe(
retryWhen(
genericRetryStrategy({
retryMax: 3,
delayTime: 1000,
}),
),
);

const obserable$ = from(chunks).pipe(concatMap(chunkIds => reuqest(chunkIds)));

return obserable$;
}

const getWordsByVocab = (word: string) => {
return axios({
url: v3('/wordsapp/words/vocab'),
params: {
word,
},
}).then(res => {
return bayDecode(res.data.data);
});
};

const fetchOneVocabSenses = (vid: string) => {
return axios({
url: v3('/wordsapp/words/vocab_senses'),
params: {
vocab_ids: vid,
},
}).then(res => {
const data = _.get(bayDecode(res.data.data), 'objects[0]', []);
return data.senses.map((i: any) => ({
id: i.id,
pos: i.pos,
cn: i.definition_cn,
// en: i.definition_en,
}));
});
};

export const wordsServices = {
genVocabDetailStream,
getVocabSenses,
getVocabExamples,
getExtExamples,
getVocabsDetailByIds,
fetchVocabIdsInfo$,
getWordsByVocab,
fetchOneVocabSenses,
};

export interface IVocabDetail {
id: string;
sense: {
word: string;
vocabulary_id: string;
senses: any[];
sound: any;
[key: string]: any;
};
examp: any[];
extExample: any[];
affixe:
| {
word_branch: any[];
word_tree: any[];
[key: string]: any;
}
| undefined;
collinsSense:
| {
cn: any[];
en: any[];
}
| undefined;
collinsExample: any[];
roots: object | undefined;
notes?: {
notes: any[];
user: {
[key: string]: any;
};
};
}