Configure Restbase to work with the Math extension

To install mathoid and restbase to work with the math extension the following steps are required:

Mathoid

npm i mathoid cd node_modules/mathoid

edit config.yaml as follows

# Number of worker processes to spawn. # Set to 0 to run everything in a single process without clustering. # Use 'ncpu' to run as many workers as there are CPU units num_workers: ncpu # Log error messages and gracefully restart a worker if v8 reports that it # uses more heap (note: not RSS) than this many mb. worker_heap_limit_mb: 500 # Logger info logging:  level: warn # Statsd metrics reporter metrics:  type: log  #host: localhost  #port: 8125 services:  - name: mathoid  # a relative path or the name of an npm package, if different from name  module: ./app.js  # optionally, a version constraint of the npm package  # version: ^0.4.0  # per-service config  conf:  port: 10042  # interface: localhost # uncomment to only listen on localhost  # more per-service config settings  svg: true  img: true  png: true  texvcinfo: true  speech: true  speech_on: false  chem: true  no_check: true

run mathoid as a screen session

screen node server.js -c config.yaml

testing mathoid

Now the following test

curl -d 'q=E=mc^2' localhost:10042/mml

should return

<math xmlns="http://www.w3.org/1998/Math/MathML" display="block">  <semantics>  <mrow>  <mi>E</mi>  <mo>=</mo>  <mi>m</mi>  <msup>  <mi>c</mi>  <mn>2</mn>  </msup>  </mrow>  <annotation encoding="application/x-tex">E=mc^2</annotation>  </semantics> </math>

RESTBase

npm i restbase cd node_modules/restbase

edit config.yaml as follows

paths:  /{api:v1}:  x-modules:  # swagger options, overriding the shared ones from the merged specs (?)  - spec:  info:  version: 1.0.0-beta  title: Formulasearchengine REST API  description: >  This API aims to provide a preview of new features reagarding  Math rendering and search, supposed to be integrated to  the [wikimedia restbase api](https://wikimedia.org/api/rest_v1/?doc).  ### High-volume access  - Don't perform more than 500 requests/s to this API.  - Set a unique `User-Agent` header that allows us to contact you  quickly. Email addresses or URLs of contact pages work well.  termsOfService: https://wikimediafoundation.org/wiki/Terms_of_Use  contact:  name: the Wikimedia Services team  url: http://mediawiki.org/wiki/RESTBase  license:  name: Apache2  url: http://www.apache.org/licenses/LICENSE-2.0  securityDefinitions: &wp/content-security/1.0.0  mediawiki_auth:  description: Checks permissions using MW api  type: apiKey  in: header  name: cookie  x-internal-request-whitelist:  - /http:\/\/[a-zA-Z0-9\.]+\/w\/api\.php/  # Override the base path for host-based (proxied) requests. In our case,  # we proxy https://{domain}/api/rest_v1/ to the API.  x-host-basePath: /api/rest_v1  x-route-filters:  - path: ./lib/normalize_title_filter.js  options:  redirect_cache_control: '{{options.purged_cache_control}}'  paths:  /media:  x-modules:  - path: v1/mathoid.yaml  options: '{{options.mathoid}}'  /testing:  x-modules:  - path: ../mathpipe.yaml  # options: '{{options.mathoid}}'  options: '{{options}}'  /{api:sys}:  x-modules:  - spec:  paths:  /mathoid:  x-modules:  - path: sys/mathoid.js  options: '{{options.mathoid}}'  /table:  x-modules:  - path: sys/table.js  options:  conf: '{{options.table}}'  /key_value:  x-modules:  - path: sys/key_value.js  /key_rev_value:  x-modules:  - path: sys/key_rev_value.js  /key_rev_latest_value:  x-modules:  - path: sys/key_rev_latest_value.js  /post_data: &sys_post_data  x-modules:  - path: sys/post_data.js  /events:  x-modules:  - path: sys/events.js  options: '{{options.events}}'  options: '{{options}}'

run restbase as a screen session

screen node server.js -c config.yaml

testing restbase

Navigate to http://localhost:8081/wikimedia.org/v1/?doc you should be able to enter a texvc formula to the check command POST endpoing i.e. /media/math/check/{type} (See screenshot 1)[1] In the response headers you should find something like

"x-resource-location": "4c0004393a88f350a93bcef62106d556c7fc827b"

If you copy the value 4c0004393a88f350a93bcef62106d556c7fc827b and paste it to the GET endpoint /media/math/render/{format}/{hash} as value you should be able to see the PNG image if you select that as an output format. (See screenshot 2)[2]

Configuration Example for arbitrary domain in RESTBase setup

If you want to use domain other than wikimedia.org, you can follow below example;

mathoid/config.yaml

# Number of worker processes to spawn. # Set to 0 to run everything in a single process without clustering. # Use 'ncpu' to run as many workers as there are CPU units num_workers: 0 # Log error messages and gracefully restart a worker if v8 reports that it # uses more heap (note: not RSS) than this many mb. worker_heap_limit_mb: 250 # Logger info logging:  level: trace # streams: # # Use gelf-stream -> logstash # - type: gelf # host: logstash1003.eqiad.wmnet # port: 12201 # Statsd metrics reporter metrics:  #type: log  #host: localhost  #port: 8125 services:  - name: mathoid  # a relative path or the name of an npm package, if different from name  module: ./app.js  # optionally, a version constraint of the npm package  # version: ^0.4.0  # per-service config  conf:  port: 10044  # interface: localhost # uncomment to only listen on localhost  # more per-service config settings  # the location of the spec, defaults to spec.yaml if not specified  # spec: ./spec.template.yaml  # allow cross-domain requests to the API (default '*')  #cors: '*'  # to disable use:  # cors: false  # to restrict to a particular domain, use:  # cors: restricted.domain.org  # content for the CSP headers  # csp: false # uncomment this line to disable sending them  # URL of the outbound proxy to use (complete with protocol)  # proxy: http://my.proxy.org:8080  # the list of domains for which not to use the proxy defined above  # no_proxy_list:  # - domain1.com  # - domain2.org  # the list of incoming request headers that can be logged; if left empty,  # the following headers are allowed: cache-control, content-length,  # content-type, if-match, user-agent, x-request-id  # log_header_whitelist:  # - cache-control  # - content-length  # - content-type  # - if-match  # - user-agent  # - x-request-id  # list of enabled renders  svg: true  img: true  png: true #new feature  speech: true #new feature  texvcinfo: true  speech_on: true  no_check: false  dpi: 180  svgo: false  # the user agent to use when issuing requests  # user_agent: service-template-node  # the template used for contacting the MW API  #mwapi_req:  # method: post  # uri: https://{{domain}}/w/api.php  # headers:  # user-agent: '{{user-agent}}'  # body: '{{ default(request.query, {}) }}'  # the template used for contacting RESTBase  #restbase_req:  # method: '{{request.method}}'  # uri: https://{{domain}}/api/rest_v1/{+path}  # query: '{{ default(request.query, {}) }}'  # headers: '{{request.headers}}'  # body: '{{request.body}}'

restbase/config.yaml

services:  - name: restbase  module: hyperswitch  conf:  port: 7231  salt: secret  default_page_size: 125  user_agent: RESTBase  ui_name: RESTBase  ui_url: https://www.mediawiki.org/wiki/RESTBase  ui_title: RESTBase docs  spec:  x-request-filters:  - path: lib/security_response_header_filter.js  x-sub-request-filters:  - type: default  name: http  options:  allow:  - pattern: http://localhost/api.php  forward_headers: true  - pattern: http://localhost:8000  forward_headers: true  - pattern: http://localhost:10044  forward_headers: true  - pattern: /^https?:\/\//  paths:  /{domain:localhost}: &arbitary_domain  x-modules:  - path: projects/MR.yaml  options:  action:  # XXX Check API URL!  apiUriTemplate: http://localhost/api.php  # XXX Check the base RESTBase URI  baseUriTemplate: "{{'http://localhost:7231/{domain}/v1'}}"  parsoid:  # XXX Check Parsoid URL!  host: http://localhost:8000  mathoid:  # XXX Check Mathoid URL!  host: http://localhost:10044  table:  backend: sqlite  dbname: db.sqlite3  pool_idle_timeout: 20000  retry_delay: 250  retry_limit: 10  show_sql: false  /{domain:your.some.domain}: *arbitary_domain # Finally, a standard service-runner config. info:  name: restbase logging:  name: restbase  level: info

restbase/project/MR.yaml

# will work with Mathoid on any {domain} paths:  /{api:v1}:  x-modules:  # swagger options, overriding the shared ones from the merged specs (?)  - spec:  info:  version: 1.0.0  title: Wikimedia REST API  description: Welcome to your RESTBase API.  #securityDefinitions: &wp/content-security/1.0.0  # mediawiki_auth:  # description: Checks permissions using MW api  # type: apiKey  # in: header  # name: cookie  # x-internal-request-whitelist:  # - /http:\/\/[a-zA-Z0-9\.]+\/w\/api\.php/  # Override the base path for host-based (proxied) requests. In our case,  # we proxy https://{domain}/api/rest_v1/ to the API.  #x-host-basePath: /api/rest_v1  x-route-filters:  - path: ./lib/normalize_title_filter.js  options:  redirect_cache_control: '{{options.purged_cache_control}}'  paths:  /media:  x-modules:  - path: v1/mathoid.yaml  options: '{{options.mathoid}}'  #/testing:  # x-modules:  # - path: ../mathpipe.yaml  # options: '{{options.mathoid}}'  /page:  x-modules:  - path: v1/content.yaml  options:  purged_cache_control: '{{options.purged_cache_control}}'  - path: v1/common_schemas.yaml # Doesn't really matter where to mount it.  /transform:  x-modules:  - path: v1/transform.yaml  options: '{{options}}'  /{api:sys}:  x-modules:  - spec:  paths:  /mathoid:  x-modules:  - path: sys/mathoid.js  options: '{{options.mathoid}}'  /table:  x-modules:  - path: sys/table.js  options:  conf: '{{options.table}}'  /key_value:  x-modules:  - path: sys/key_value.js  /key_rev_value:  x-modules:  - path: sys/key_rev_value.js  /key_rev_latest_value:  x-modules:  - path: sys/key_rev_latest_value.js  /post_data: &sys_post_data  x-modules:  - path: sys/post_data.js  /events:  x-modules:  - path: sys/events.js  options: '{{options.events}}'  /page_revisions:  x-modules:  - path: sys/page_revisions.js  /action:  x-modules:  - path: sys/action.js  options: '{{options.action}}'  /page_save:  x-modules:  - path: sys/page_save.js  /parsoid:  x-modules:  - path: sys/parsoid.js  options:  parsoidHost: '{{options.parsoid.host}}'  response_cache_control: '{{options.purged_cache_control}}'  options: '{{options}}'

restbase/v1/mathoid.yaml

# Mathoid - math formula rendering service :simply replaced /wikimedia.org/sys to /{domain}/sys tags:  - name: Math  description: formula rendering paths:  /math/check/{type}:  post:  tags: ['Math']  summary: Check and normalize a TeX formula.  description: |  Checks the supplied TeX formula for correctness and returns the  normalised formula representation as well as information about  identifiers. Available types are tex and inline-tex. The response  contains the `x-resource-location` header which can be used to retrieve  the render of the checked formula in one of the supported rendering  formats. Just append the value of the header to `/media/math/{format}/`  and perform a GET request against that URL.  Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).  produces:  - application/json  parameters:  - name: type  in: path  description: The input type of the given formula; can be tex or inline-tex  type: string  required: true  enum:  - tex  - inline-tex  - chem  - name: q  in: formData  description: The formula to check  type: string  required: true  responses:  '200':  description: Information about the checked formula  '400':  description: Invalid type  schema:  $ref: '#/definitions/problem'  default:  description: Error  schema:  $ref: '#/definitions/problem'  x-monitor: true  x-amples:  - title: Mathoid - check test formula  request:  params:  domain: wikimedia.org  type: tex  body:  q: E=mc^{2}  response:  status: 200  headers:  content-type: /^application\/json/  x-resource-location: /.+/  cache-control: 'no-cache'  body:  success: true  checked: /.+/  x-request-handler:  - get_from_sys:  request:  method: post  uri: /{domain}/sys/mathoid/check/{type}  headers: '{{ request.headers }}'  body: '{{ request.body }}'  /math/formula/{hash}:  get:  tags: ['Math']  summary: Get a previously-stored formula  description: |  Returns the previously-stored formula via `/media/math/check/{type}` for  the given hash.  Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).  produces:  - application/json  parameters:  - name: hash  in: path  description: The hash string of the previous POST data  type: string  required: true  minLength: 1  responses:  '200':  description: Information about the checked formula  '404':  description: Data for the given hash cannot be found  schema:  $ref: '#/definitions/problem'  default:  description: Error  schema:  $ref: '#/definitions/problem'  x-monitor: false  x-request-handler:  - get_from_sys:  request:  method: get  uri: /{domain}/sys/mathoid/formula/{hash}  headers: '{{ request.headers }}'  /math/render/{format}/{hash}:  get:  tags: ['Math']  summary: Get rendered formula in the given format.  description: |  Given a request hash, renders a TeX formula into its mathematic  representation in the given format. When a request is issued to the  `/media/math/check/{format}` POST endpoint, the response contains the  `x-resource-location` header denoting the hash ID of the POST data. Once  obtained, this endpoint has to be used to obtain the actual render.  Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).  produces:  - image/svg+xml  - application/mathml+xml  - image/png  parameters:  - name: format  in: path  description: The output format; can be svg or mml  type: string  required: true  enum:  - svg  - mml  - png  - name: hash  in: path  description: The hash string of the previous POST data  type: string  required: true  minLength: 1  responses:  '200':  description: The rendered formula  '404':  description: Unknown format or hash ID  schema:  $ref: '#/definitions/problem'  default:  description: Error  schema:  $ref: '#/definitions/problem'  x-monitor: false  x-setup-handler:  - init_svg:  uri: /{domain}/sys/key_value/mathoid.svg  body:  keyType: string  valueType: string  - init_mml:  uri: /{domain}/sys/key_value/mathoid.mml  body:  keyType: string  valueType: string  - init_png:  uri: /{domain}/sys/key_value/mathoid.png  body:  keyType: string  valueType: blob  x-request-handler:  - check_storage:  request:  method: get  uri: /{domain}/sys/key_value/mathoid.{$.request.params.format}/{$.request.params.hash}  headers:  cache-control: '{{ cache-control }}'  catch:  status: 404  return_if:  status: '2xx'  return:  status: 200  headers: "{{ merge({ 'cache-control': options.cache-control }, check_storage.headers) }}"  body: '{{ check_storage.body }}'  - postdata:  request:  uri: /{domain}/sys/mathoid/formula/{request.params.hash}  - mathoid:  request:  method: post  uri: /{domain}/sys/mathoid/render/{request.params.format}  headers:  content-type: application/json  x-resource-location: '{{ request.params.hash }}'  body: '{{postdata.body}}'

Apache proxy config : /etc/httpd/conf.d/restbase.conf

# # Restbase # <VirtualHost *:80>  ProxyPreserveHost On  ProxyPass /your.some.domain/ http://localhost:7231/your.some.domain/  ProxyPassReverse /your.some.domain/ http://localhost:7231/your.some.domain/  ProxyPass /localhost/ http://localhost:7231/localhost/  ProxyPassReverse /localhost/ http://localhost:7231/localhost/  # MediaWiki is located here:  DocumentRoot /var/www/html  </VirtualHost>

you can test the APIs at http://your.some.domain/your.some.domain/v1/

setup apache proxy

To load the images from the same location where the main wiki site is hosted a simple apache proxy can be used. Enable apache mod proxy ... and then add a config file along the lines of this file

cat /etc/apache2/sites-available/restbase.conf

<VirtualHost *:80>  ServerName restbase.formulasearchengine.com  ServerAlias api.formulasearchengine.com png.formulasearchengine.com  ServerAdmin wiki@physikerwelt.de  ProxyPreserveHost On  ProxyPass /v1/ http://localhost:8081/wikimedia.org/v1/  ProxyPass /wikimedia.org/v1/ http://localhost:8081/wikimedia.org/v1/ </VirtualHost>

a2ensite restbase.conf

You can use letsencrypt to also generate a HTTPS version for that site.

Helpful commands to check the state of the database

Clean cache

run cqlsh

use "local_group_default_T_mathoid_svg"; truncate data; truncate meta; use "local_group_default_T_mathoid_mml"; truncate data; truncate meta; use "local_group_default_T_mathoid_input"; truncate data; truncate meta; use "local_group_default_T_mathoid_png"; truncate data; truncate meta;

Browse the data

Task retrieve the input given a key ... in this example we'll use 0f7fa7bb7d214c60f8d2f0b8d356c41bfe5c5d1d as key run cqlsh

use "local_group_test_T_mathoid_input"; SELECT * from data WHERE key = '0f7fa7bb7d214c60f8d2f0b8d356c41bfe5c5d1d' and "_domain" = 'wikimedia.org';

 _domain | key | tid | _del | content-location | content-sha256 | content-type | headers |latestTid | tags | value ---------------+------------------------------------------+--------------------------------------+------+------------------+----------------+------------------+-------------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------- wikimedia.org | 0f7fa7bb7d214c60f8d2f0b8d356c41bfe5c5d1d | e736aba4-9edc-11e5-927a-301f88a60cb9 | null | null | null | application/json | {"content-type":"application/json"} | null | null | {"q":"=nf(x)\\left({n-1 \\choose k-1}F(x)^{k-1}(1-F(x))^{(n-1)-(k-1)}-\\underbrace {{n-1 \\choose n}F(x)^{n}(1-F(x))^{(n-1)-n}} \\right)","type":"inline-tex"}