gianlp.models.text_representations.per_chunk_sequencer.PerChunkSequencer

class gianlp.models.text_representations.per_chunk_sequencer.PerChunkSequencer(sequencer: TextRepresentation, chunker: Callable[[str], List[str]], chunking_maxlen: int)

Bases: TextRepresentation

Per chunk sequencer wrapper For each chunk creates a sequence using the text input provided

Variables

• _chunker – function used for chunking the texts

• _sequencer – text input use for sequencing each chunk

• _chunking_maxlen – the maximum length in chunks for a text

Parameters

• sequencer – TextInput object

• chunker – function for chunking texts

• chunking_maxlen – the maximum length in chunks for a text

Methods

build	Builds the whole chain of models in a recursive manner using the functional API
deserialize	Deserializes a model
parallel_tokenizer	Parallelizable wrapper for the tokenizer
preprocess_texts	Given texts returns the array representation needed for forwarding the keras model
serialize	Serializes the model to be deserialized with the deserialize method
tokenize_texts	Function for tokenizing texts

Attributes

inputs	Method for getting all models that serve as input.
inputs_shape	Returns the shapes of the inputs of the model
outputs_shape	Returns the output shape of the model
trainable_weights_amount	Computes the total amount of trainable weights
weights_amount	Computes the total amount of weights

preprocess_texts(texts: Union[List[str], Series]) → Union[List[ndarray], ndarray]

Given texts returns the array representation needed for forwarding the keras model

Parameters

texts – the texts to preprocess

Returns

a numpy array of shape (#texts, _sequence_maxlen)

build(texts: Union[List[str], Series]) → None

Builds the whole chain of models in a recursive manner using the functional API

Parameters

texts – the texts for building if needed

classmethod deserialize(data: bytes) → BaseModel

Deserializes a model

Parameters

data – the data for deserializing

Returns

a BaseModel object

static get_bytes_from_model(model: Model, copy: bool = False) → bytes

Transforms a keras model into bytes

Parameters

• model – the keras model

• copy – whether to copy the model before saving. copying the model is needed for complex nested models because the keras save/load can fail

Returns

a byte array

static get_model_from_bytes(data: bytes) → Model

Given bytes from keras model serialized with get_bytes_from_model method returns the model

Parameters

data – the model bytes

Returns

a keras model

property inputs: ModelInputsWrapper

Method for getting all models that serve as input. All TextRepresentation have no models as an input.

Returns

a list or list of tuples containing BaseModel objects

property inputs_shape: Union[List[ModelIOShape], ModelIOShape]

Returns the shapes of the inputs of the model

Returns

a list of shape tuple or shape tuple

property outputs_shape: ModelIOShape

Returns the output shape of the model

Returns

a list of shape tuple or shape tuple

static parallel_tokenizer(text: str, tokenizer: Callable[[str], List[str]], sequence_maxlength: Optional[int] = None) → List[str]

Parallelizable wrapper for the tokenizer

Parameters

• text – the text to tokenize

• tokenizer – the tokenizer

• sequence_maxlength – optional sequence maxlength.

Returns

a list of lists with string tokens

serialize() → bytes

Serializes the model to be deserialized with the deserialize method

Returns

a byte array

static tokenize_texts(texts: Union[List[str], Series], tokenizer: Callable[[str], List[str]], sequence_maxlength: Optional[int] = None) → List[List[str]]

Function for tokenizing texts

Parameters

• texts – the texts to tokenize

• tokenizer – the tokenizer

• sequence_maxlength – optional sequence maxlength.

Returns

a list of lists with string tokens

property trainable_weights_amount: Optional[int]

Computes the total amount of trainable weights

Returns

the total amount of trainable weights or none if not built

property weights_amount: Optional[int]

Computes the total amount of weights

Returns

the total amount of weights or none if not built