Make markdown Grammar¶

Token¶

class mpiece.lexer.Token(render_func, text=None, extras={}, order=[], extras_to_children={})¶

The token save info about markdown grammar. They are used in the parse function in lexer class.

Parameters:

render_func (str) –
Render function name which the token will use.

Note

All render function name start with the render_ prefix. You shoul add the function name without include this prefix.
text (str) – Text inside of the markdown grammar. This text will be re-parsed to find more markdown grammar.
extras (dict) – Extra data to render function. The dictionary elements will convert in arguments for the render function
order ([str]) – Markdown grammar elements that the lexer will find in the token text.
extras_to_children (dict) – Extra data used in the children parse functions.

Grammar order¶

This diagram show the order of the lexer to time parse the markdown grammar. It is important if you want add new grammar.

mpiece.lexer.Lexer has 3 main order attrs:

mpiece.lexer.Lexer.order_block
mpiece.lexer.Lexer.order_inline
mpiece.lexer.Lexer.order_initial

When you make a new grammar you should modify the one of this attributes for add the new grammar to the lexer.

Setps to make a new grammar¶

To make new grammar for our markdown lexer is necessary 5 steps:

Define a unique name.

Make a regular expression.

Make a parser function in our lexer class.

Define the order of our grammar in the lexer class.

Make a render function in our Renderer class.

Make new grammar. Example 1¶

Objective¶

Make a markdown grammar to change the color in a text string.

Normal string. #f00 String with different color.##
Normal string.
#0000dd Another string with different color.##
#1b70b0 **bold text**, *italic text*, _underline text_, ~strike text~, `no code inline text`.##

This markdown text will transform in

<p>
        Normal string. <span style="color:#f00">String with different color.</span> Normal string.
        <span style="color:#0000dd">Another string with different color.</span>
        <span style="color:#1b70b0">
                <strong>bold text</strong>, <em>italic text</em>, <ins>underline text</ins>, <del>strike text</del>,
                `no code inline text`
        </span>
</p>

1. Define a name¶

The name for this examples is color

2. Make the regular expression¶

The regular expression for the color grammar

#(?P<color>[0-9a-fA-F]{3}|[0-9a-fA-F]{6}) (?P<text>.+?)##

Making a custom lexer class and add the regular expression.

from mpiece.lexer import Lexer
import re

class CustomLexer(Lexer):
    # The regular expression
    regex_color = re.compile(r'#(?P<color>[0-9a-fA-F]{3}|[0-9a-fA-F]{6}) (?P<text>.+?)##')

The name of the regular expression in the lexer class should be regex_ prefix + the name selected. In this case regex_color

3. Make the parse function¶

The parse function process the result of the python function re.sub in our regular expression.

from mpiece.lexer import Lexer, Token
import re

class CustomLexer(Lexer):
    # The regular expression
    regex_color = re.compile(r'#(?P<color>[0-9a-fA-F]{3}|[0-9a-fA-F]{6}) (?P<text>.+?)##')

    def parse_color(self, mo):
        """ mo is a python Match object.
            mo is the result of ``re.sub`` using the regular expression ``regex_color``.

            This parse function always should return:
                    - Token object
                    - string
                    - List|tuple of objects and/or strings.
        """
        color = mo.group('color')
        text = mo.group('text')
        order = ['bold', 'italic', 'underline', 'strike']

        # 'color': Is the name of the render function witouth the prefix ``render_``
        # text: The text obtained in the color grammar. This text will be re-parsed to finding more markdown grammar.
        # extras: Extra arguments for the render function.
        # order: Grammar elements which the lexer will search inside of the text.
        return Token('color', text, extras={'color': color}, order=order)

In the example the text inside of the color grammar will be re-parsed finding the grammar of the variable order. The parse function name should be parse_ prefix + the name selected. In this case parse_color.

4. Define order¶

Define the order using the function Lexer.define_order() method.

from mpiece.lexer import Lexer, Token
import re

class CustomLexer(Lexer):
    # The regular expression
    regex_color = re.compile(r'#(?P<color>[0-9a-fA-F]{3}|[0-9a-fA-F]{6}) (?P<text>.+?)##')

    # Define the order and insert the color grammar.
    def define_order(self):
            # add color grammar to the ``order_inline`` variable.
            # it is important doing before to call the father function.
        self.order_inline.insert(1, 'color')
        # call the father function.
        super(CustomLexer, self).define_order()

    def parse_color(self, mo):
        """ mo is a python Match object.
            mo is the result of ``re.sub`` using the regular expression ``regex_color``.

            This parse function always should return:
                    - Token object
                    - string
                    - List|tuple of objects and/or strings.
        """
        color = mo.group('color')
        text = mo.group('text')
        order = ['bold', 'italic', 'underline', 'strike']

        # 'color': Is the name of the render function witouth the prefix ``render_``
        # text: The text obtained in the color grammar. This text will be re-parsed to finding more markdown grammar.
        # extras: Extra arguments for the render function.
        # order: Grammar elements which the lexer will search inside of the text.
        return Token('color', text, extras={'color': color}, order=order)

5. Make renderer function¶

We want convert the markdown text in html code, so, we will use the HtmlRenderer class as the father of our renderer class.

from mpiece.lexer import Lexer, Token
from mpiece.renderer import HtmlRenderer
import re

class CustomLexer(Lexer):
    # The regular expression
    regex_color = re.compile(r'#(?P<color>[0-9a-fA-F]{3}|[0-9a-fA-F]{6}) (?P<text>.+?)##')

    # Define the order and insert the color grammar.
    def define_order(self):
            # add color grammar to the ``order_inline`` variable.
            # it is important doing before to call the father function.
        self.order_inline.insert(1, 'color')
        # call the father function.
        super(CustomLexer, self).define_order()

    def parse_color(self, mo):
        """ mo is a python Match object.
            mo is the result of ``re.sub`` using the regular expression ``regex_color``.

            The parse function always should return:
                    - Token object
                    - string
                    - List|tuple of objects and/or strings.
        """
        color = mo.group('color')
        text = mo.group('text')
        order = ['bold', 'italic', 'underline', 'strike']

        # 'color': Is the name of the render function witouth the prefix ``render_``
        # text: The text obtained in the color grammar. This text will be re-parsed to finding more markdown grammar.
        # extras: Extra arguments for the render function.
        # order: Grammar elements which the lexer will search inside of the text.
        return Token('color', text, extras={'color': color}, order=order)

class CustomRenderer(HtmlRenderer):

    # Render function for the color grammar.
    def render_color(self, text, color):
            # convert html characters
            text = self.escape(text)
            return '<span class="color:#%s">%s</span>' % (color, text)

With this code, the example is complete. Now you can use this new lexer and renderer to convert markdown text in html code.

Make new grammar example 2¶

Objective¶

Add emojis in the markdown text.

Markdown text :emoji1: continue :XD: continue :not_exists: end.

<p>
    Markdown text <span class="css-emoji1"></span> continue <span class="emojis-emoji-smiley"></span>
   :not_exists: end.
</p>

1. Define a name¶

The name for this examples is emojis

2. Make the regular expression¶

The regular expression for this example:

:(?P<emoji>[a-zA-Z0-9_]+):

Now starting making the lexer.

from mpiece.lexer import Lexer
import re


class EmojiLexer(Lexer):
    regex_emojis = re.compile(r':(?P<emoji>[a-zA-Z0-9_]+):')

3. Make the parse function¶

from mpiece.lexer import Lexer, Token
import re


class EmojiLexer(Lexer):
    # emoji list
    emoji_list = {
        'emoji1': 'css-emoji1',
        'XD': 'emojis-emoji-smiley'
    }


    regex_emojis = re.compile(r':(?P<emoji>[a-zA-Z0-9_]+):')

    def parse_emojis(self, mo):
        emoji = mo.group('emoji')
        emoji_css = self.emoji_list.get(emoji, False)

        # check if the emoji exists
        if not emoji_css:
            # emoji doesn't exits. return the same string
            return ':%s:' % emoji

        # emoji exists
        return Token(extras={'emoji_css_class': emoji_css}, order=[])

4. Define order¶

Emoji grammar is added to the end of order_inline.

from mpiece.lexer import Lexer, Token
import re


class EmojiLexer(Lexer):
    # emoji list
    emoji_list = {
        'emoji1': 'css-emoji1',
        'XD': 'emojis-emoji-smiley'
    }


    regex_emojis = re.compile(r':(?P<emoji>[a-zA-Z0-9_]+):')

    def define_order(self):
        self.order_inline.append('emojis')
        super(EmojiLexer, self).define_order()

    def parse_emojis(self, mo):
        emoji = mo.group('emoji')
        emoji_css = self.emoji_list.get(emoji, False)

        # check if the emoji exists
        if not emoji_css:
            # emoji doesn't exits. return the same string
            return ':%s:' % emoji

        # emoji exists
        return Token(extras={'emoji_css_class': emoji_css}, order=[])

5. Make renderer function¶

from mpiece.lexer import Lexer, Token
from mpiece.renderer import HtmlRenderer
import re


class EmojiLexer(Lexer):
    # Private attribute.
    # list with all emojis and his css class
    emoji_list = {
            'emoji1': 'css-emoji1',
            'XD': 'emojis-emoji-smiley'
    }

    regex_emojis = re.compile(r':(?P<emoji>[a-zA-Z0-9_]+):')

    def define_order(self):
        self.order_inline.append('emojis')
        super(EmojiLexer, self).define_order()

    def parse_emojis(self, mo):
        emoji = mo.group('emoji')
        emoji_css = self.emoji_list.get(emoji, False)

        # check if the emoji exists
        if not emoji_css:
            # emoji doesn't exits. return the same string
            return ':%s:' % emoji

        # emoji exists
        return Token('emojis', extras={'emoji_css_class': emoji_css}, order=[])


class EmojiRenderer(HtmlRenderer):
    def render_emojis(self, emoji_css_class):
        return '<span class="%s"></span>' % emoji_css_class

Make markdown Grammar¶

Token¶

Grammar order¶

Setps to make a new grammar¶

Make new grammar. Example 1¶

Objective¶

1. Define a name¶

2. Make the regular expression¶

3. Make the parse function¶

4. Define order¶

5. Make renderer function¶

Make new grammar example 2¶

Objective¶

1. Define a name¶

2. Make the regular expression¶

3. Make the parse function¶

4. Define order¶

5. Make renderer function¶

Table Of Contents

Previous topic

Next topic

This Page