扇贝单词前端 JavaScript 逆向

在写扇贝单词的 Chrome 插件中，需要使用一些非公开的 API，在此记录一下遇到的波折。

插件 Github 链接：https://github.com/zerolfx/shanbay-helper

由于需要完成任意网页查词的功能（扇贝只支持自己网页内的查词），所以要先观察一下，在扇贝网页内查词时调用了什么 API。借助 Chrome 的开发者工具中的 Network，很容易定位到这个请求。这是一个 GET 请求，请求的 URL 为 https://apiv3.shanbay.com/wordsapp/words/vocab?word=reverse（以单词 reverse 为例），返回的 JSON 信息如下。

{"data": "T5SXR44XQRUOCNXHCXNQYAP22AOCMSOCCFTGYAPMNBYAP256CO4XOCNXHCXNQ4XOCCFOCORIH2AGEORXYATSTGX2AGEOPP2TT7IEZXHZ2PTGR44XQRUIHK363P2CFURYA2EYABYAIHORCPVZURJX56G32LP256CFWUVTSURHZVMHZCFWUVTSUREWZOTSIH2AURYAPMNBYAGJCFA7QCFTWVMG2XHZ2PTGX2AQIHP2SCB4XGEEWL2AP2ORCPVZURJXVZG3VZCPVZURJQEZURWURYAPMNBYAG3CFVMEZGJTTCO63P2CFGE63PEEW3OKOC2LXHCXNQEWURKZO252EJL25URHZIHTS2EKZORMOCEWXTWURRMAEWUR56VMYA2ESCG3MSOCHZZO4XURSCGEEWGENXSNXNQ7QP2TT7TRUPEVZCOVZGJVZCOU3G3TTVZORXHZ2PTGXHZOCHZGEHZZORMGEEWXRMGETT3EWOCHZXTWURJQOR2EJXTSURRMG3CPGEHZQYAGEKUVRM2EJ3ORXYATSTGXEW3VZOCKOROPCFTWVMRMCFTWVM4XPEWURYA2EYAAPEXEW7Q25P2WARM2EYA4VOPPETTVMEZGJTTLOKG3CFVMEZGJTT4VCPPMEWQ7QR4TT7TU3PMEWGEOPPE2LCO7QG3EWGE63IHWVMYACF563VZOCKOROPPMRMLMSGERMBTSURHZTGTWOCEWVMHZUR56QHZ2ESCZOTSURKUV7IGETTUR56URHZ7QEWG3RMBTSURRMZOTSPMHZ7TYAOCSCL56OCKXTWG3RMQHZGEHZXVZGERMGEEWGERM37QGEJZORMGESC7TORGEJUV56URRMUVCPPMEWVZTSUR4XXSNXNQ2PP2W7TTSIHRM2POPPM56VZOROCKOR7QPMTT3VZOCKOROPUR2L7IYAG3CFOREZPEYA7IHZPE5625OPG3TTQHZCFTWAVZG3ORCO7QP2TT7TRUPE4X4VCP2EJIHTSURJXMSGE56OCORG3RMGEYAG3RMEZ25URJAOR2ESCVMHZGEHZXMSOCEWURTSURJUR25URNMS25G3EWUVCPGEEW7TYAGE56UR4XG3RMG34XGESCIHRMOCHZIHCPG3SCZO25URRMUV25OCSCZOVZGEHZURVZUR2L7IOKIHJURYACF2LTSTGXEW3VZOCKOROPCFTWVMG2CFTWVM4XPEWURYA2EYAAPEXEW7Q25P2WARM2EYA4VOPPETTVMEZGJTTLOKG3CFVMEZGJTT4VCPPMEWQ7QR4TT7TU3PMEWGEOPPE2LCO7QG3EWGE63IHWVMYACF563VZOCKOROPPMRMUVMSURRMQYAG3RMX56URHZVMEWURJZOCPOCHZL25GEHZ37QGEKUR56URTTX4XOCSCZOCPG3EWVMEWPMEW7TORUREWVMEWOCEWG356G3SC7TYAGETT3YAGEHZXMSOCTTGEYAG3RMG3VZURJL4XURHZIHRMG3SCTGMSPMEWVZTSUR4XXSNXNQ2PP2W7TTSIHRM2POPPM56VZOROCKOR7QPMTT3VZOCKOROPUR2L7IYAG3CFOREZPEYA7IHZPE5625OPG3TTQHZCFTWAVZG3ORCO7QP2TT7TRUPE4X4VVZ2EJUR4XG3EWUR4XGEHZXVZOCHZB25URTTG3CPGEJG3CPG3SC7THZGEHZ3YAUREWUV25URTTQOROCYA7IEZOCSCQOROCEWOCEWGEEWL25G3HZVM7QG3HZG34X2EKVMHZG3EWUR56GESCBCPURHZXTWUR56LMS2EN7IOKIHJURYACFCF25SNXNQHZIHEWVM7QP2KVMEZCF56325XHZ2PTGXHZXTSURSCTGOKURJLOKURSCAUVURSCZOHCURJZOHCGESCURU32EJLMSURRMIH56IZRMBTS2EHZBTSXYATSTGX2AVMTSOCK325OCTT7T63G3CFZOYA2EYABYAURHZBCP2E2L25TSUR4X25CPGE3ZOTS2EJ2PRM2ESC2PRM2ENMSRMGEJEZ56URRMEZG2URJBHCURJBYAPMNBYAIHEWVMEWCF56OREZXHZ2PTGXEW37QG3TT37QXYATSTGX2AOCOPG3563YACFTW7T7IIHKUVYA2EYABTSPMNBYAG356COOKPETTVMU3P2NXHCXNXYA63ZO=="}

看到大小为 32 的字符集以及最后的两个等号，就知道这是 Base32 编码了。然后用这个在线工具 https://cryptii.com/pipes/base32 （这个网站支持许多能自定义的编码方式的加解密）尝试解码，然而事情并没有那么简单，转换出来的二进制并不能显示为文本。于是随手试了试把 Base32 中的 A-Z 和 2-7 交换前后顺序以及内部颠倒，但还是不行。所以这肯定是定制的 Base32 的，是否单单修改了编码表也不能确认。

由于解码是在前端完成的，所以我决定不如定位到对应的 JS 代码来做解密。在开发者工具的 Source 页面，可以看到扇贝的大量前端源码（React + Typescript），而且是没有经过打包前的结果。在 assets.baydn.com/web_static/words_wordsweb/static/js/services/words.ts 中找到了所有需要的 API（这个文件的源码我会贴在最后），也在其中找到了一个叫做 bayDecode 的解码函数。然后在开发者工具的 Source 页面选择在所有文件中搜索这个函数名，定位到了这个函数的定义，代码如下。

export const bayDecode = (str: string) => {
  return JSON.parse(window.bays4.d(str));
};

可以看到有一个名叫 bays4 的全局函数做了解码的主要工作。（插一句题外话，这个函数名取得真是不错，取了 shanbay 的一部分，和 base 这个单词读音近似。）如果继续在源码中搜索 bays4，会发现除了之前那个函数的调用，只出现在了 HTML 中的 script 标签内。打开 Console，输入 bays.d，可以看到这个函数的定义，单击后能跳转到源码，果然就是嵌在 HTML 中的那个函数。

可惜的是，这个函数经过了混淆，可读性极差。但是我只需要能完成解码工作的函数，并不需要理解解码的具体过程，于是把那一大段 JS 复制到本地，用之前 API 请求来的加密信息测试了一下，果然可以使用了。最后我把那段代码嵌入到了插件的 JS 文件中，供请求 API 的时候调用。

补充：在 Source 中查看 JS 源码还是一件很痛苦的事情，但是想保存到本地的话，一次只能保存单个文件。但是在这个插件的帮助下，可以批量保存到本地，用 IDE 打开，无论是搜索还是跳转都方便了许多。

解码后结果

{
  "id": "bvvxex",
  "word": "reverse",
  "sound": {
    "ipa_us": "r\u026a'v\u025c\u02d0rs",
    "ipa_uk": "r\u026a'v\u025c\u02d0s",
    "audio_us_name": "f0d48142708d3fb730f3eb968f4213f4",
    "audio_uk_name": "6c6436b35aff2702e82036162b4e381e",
    "audio_us_urls": [
      "https://media-audio1.baydn.com/abc_pub_audio/18700287fec3bc9400e95c628fc00340.4be16db7c2c62573f71a44354e456351.mp3",
      "https://media-audio1.baydn.com/abc_pub_audio/18700287fec3bc9400e95c628fc00340.4be16db7c2c62573f71a44354e456351.mp3"
    ],
    "audio_uk_urls": [
      "https://media-audio1.baydn.com/abc_pub_audio/5832bc2625f041f1461a4c61b2e41bef.de2efff6a4b5ab628ecbc65012273a88.mp3",
      "https://media-audio1.baydn.com/abc_pub_audio/5832bc2625f041f1461a4c61b2e41bef.de2efff6a4b5ab628ecbc65012273a88.mp3"
    ]
  },
  "created_at": "2018-01-10T14:04:53.818376+00:00",
  "updated_at": "2019-03-14T08:39:38.349639+00:00",
  "ref_id": "aaaaa",
  "vocab_type": 0,
  "comment": ""
}

代码 assets.baydn.com/web_static/words_wordsweb/static/js/services/words.ts

import { v3 } from 'helpers/withPrefix';
import _ from 'lodash';
import { Observable, from, defer } from 'rxjs';
import { retryWhen, concatMap } from 'rxjs/operators';
import axios, { AxiosResponse } from 'axios';
import { genericRetryStrategy } from 'helpers/rx';
import { bayDecode } from 'helpers';
import * as services from '.';

const _decodeAndKeyBy = (key: string) => (res: AxiosResponse<any>) => {
  return _.keyBy(_.get(bayDecode(res.data.data), 'objects', []), key);
};

const getVocabSenses = (vocabIds: string[], dictId?: string) => {
  return axios({
    url: v3('/wordsapp/words/vocab_senses'),
    params: {
      vocab_ids: vocabIds.join(','),
      dict_id: dictId,
    },
  }).then(_decodeAndKeyBy('vocabulary_id'));
};

const getVocabExamples = (vocabIds: string[], dictId?: string) => {
  return axios({
    url: v3('/wordsapp/words/vocab_examples'),
    params: {
      vocab_ids: vocabIds.join(','),
      dict_id: dictId,
    },
  }).then(_decodeAndKeyBy('vocab_id'));
};

const getExtExamples = async (vocabIds: string[], dictId?: string) => {
  return await axios({
    url: v3('/wordsapp/words/ext_examples'),
    params: {
      vocab_ids: vocabIds.join(','),
      dict_id: dictId,
    },
  }).then(_decodeAndKeyBy('vocab_id'));
};

const _genDetailsPromisesByChunkIds = (
  options: GenVocabDetailStreamOptions,
  chunkIds: string[],
) => {
  const emptyPromise = Promise.resolve(undefined);
  const promises = [
    getVocabSenses(chunkIds, options.dictId),
    getVocabExamples(chunkIds, options.dictId),
    options.loadExtExamples ? getExtExamples(chunkIds, options.dictId) : emptyPromise,
    options.loadAffixes
      ? services.appletsService.getAffixesByVocabIds(chunkIds)
      : emptyPromise,
    options.loadCollins
      ? services.appletsService.getCollinsSensesByVocabIds(chunkIds)
      : emptyPromise,
    options.loadCollins
      ? services.appletsService.getCollinsExamplesByVocabIds(chunkIds)
      : emptyPromise,
    options.loadRoots
      ? services.appletsService.getRootsByVocabIds(chunkIds)
      : emptyPromise,
  ];
  return promises;
};

const getVocabsDetailByIds = async (
  options: GenVocabDetailStreamOptions,
  chunkIds: string[],
): Promise<Array<IVocabDetail>> => {
  const [
    senses,
    examples,
    extExamples,
    affixes,
    collinsSenses,
    collinsExamples,
    roots,
  ] = await Promise.all(_genDetailsPromisesByChunkIds(options, chunkIds));

  const details = chunkIds.map(vid => {
    return {
      id: vid,
      sense: _.get(senses, vid),
      examp: _.get(examples, [vid, 'examples'], []),
      extExample: _.get(extExamples, [vid, 'examples'], []),
      affixe: _.get(affixes, vid),
      collinsSense: _.get(collinsSenses, [vid, 'definitions']),
      collinsExample: _.get(collinsExamples, vid, []),
      roots: _.get(roots, vid),
      notes: undefined,
      collinsVocabType: _.get(collinsSenses, [vid, 'vocab_type']),
    };
  });

  return details;
};

const _genOneGroupVocabDetailWithRetryStream = (
  options: GenVocabDetailStreamOptions,
  chunkIds: string[],
): Observable<any> =>
  defer(() => getVocabsDetailByIds(options, chunkIds)).pipe(
    retryWhen(
      genericRetryStrategy({
        retryMax: 3,
        delayTime: 1000,
      }),
    ),
  );

interface GenVocabDetailStreamOptions {
  ids: string[];
  dictId?: string;
  loadExtExamples: boolean;
  loadAffixes: boolean;
  loadRoots: boolean;
  loadCollins: boolean;
}

const genVocabDetailStream = (options: GenVocabDetailStreamOptions): Observable<any> =>
  from(_.chunk(options.ids, 10)).pipe(
    concatMap(chunkIds => _genOneGroupVocabDetailWithRetryStream(options, chunkIds)),
  );

function fetchVocabIdsInfo$(
  idList: string[],
  func: (ids: string[]) => any,
): Observable<any> {
  const chunks = _.chunk(idList, 10);
  const reuqest = (chunkIds: string[]) =>
    defer(() => func(chunkIds)).pipe(
      retryWhen(
        genericRetryStrategy({
          retryMax: 3,
          delayTime: 1000,
        }),
      ),
    );

  const obserable$ = from(chunks).pipe(concatMap(chunkIds => reuqest(chunkIds)));

  return obserable$;
}

const getWordsByVocab = (word: string) => {
  return axios({
    url: v3('/wordsapp/words/vocab'),
    params: {
      word,
    },
  }).then(res => {
    return bayDecode(res.data.data);
  });
};

const fetchOneVocabSenses = (vid: string) => {
  return axios({
    url: v3('/wordsapp/words/vocab_senses'),
    params: {
      vocab_ids: vid,
    },
  }).then(res => {
    const data = _.get(bayDecode(res.data.data), 'objects[0]', []);
    return data.senses.map((i: any) => ({
      id: i.id,
      pos: i.pos,
      cn: i.definition_cn,
      // en: i.definition_en,
    }));
  });
};

export const wordsServices = {
  genVocabDetailStream,
  getVocabSenses,
  getVocabExamples,
  getExtExamples,
  getVocabsDetailByIds,
  fetchVocabIdsInfo$,
  getWordsByVocab,
  fetchOneVocabSenses,
};

export interface IVocabDetail {
  id: string;
  sense: {
    word: string;
    vocabulary_id: string;
    senses: any[];
    sound: any;
    [key: string]: any;
  };
  examp: any[];
  extExample: any[];
  affixe:
    | {
        word_branch: any[];
        word_tree: any[];
        [key: string]: any;
      }
    | undefined;
  collinsSense:
    | {
        cn: any[];
        en: any[];
      }
    | undefined;
  collinsExample: any[];
  roots: object | undefined;
  notes?: {
    notes: any[];
    user: {
      [key: string]: any;
    };
  };
}