lal::io::treebank_collection_processor Class Reference

Automatic processing of treebank collections. More...

#include <treebank_collection_processor.hpp>

Inheritance diagram for lal::io::treebank_collection_processor:

Public Member Functions
void	set_join_files (bool v) noexcept
	Join the resulting files into a single file.
void	set_number_threads (size_t n_threads) noexcept
	Set the number of threads.
size_t	get_num_errors () const noexcept
	Returns the number of errors that arised during processing.
const treebank_error &	get_error_type (size_t i) const noexcept
	Get the ith error.
const std::string &	get_error_treebank_filename (size_t i) const noexcept
	Get the treebank's file name where the ith error happened.
const std::string &	get_error_treebank_name (size_t i) const noexcept
	Get the treebank's name for where the ith error happened.
treebank_error	init (const std::string &main_file, const std::string &output_directory) noexcept
	Initialise the processor with a new collection.
treebank_error	process (const std::string &result_filename="") noexcept
	Process the treebank collection.
void	add_feature (const treebank_feature &fs) noexcept
	Adds a feature to the processor.
void	remove_feature (const treebank_feature &fs) noexcept
	Removes a feature from the processor.
void	set_check_before_process (bool v) noexcept
	Should the treebank file or collection be checked for errors prior to processing?
void	clear_features () noexcept
	Clear the features in the processor.
void	set_separator (char c) noexcept
	Sets the separator character.
void	set_verbosity (int k) noexcept
	Sets the level of verbosity of the process methods.
void	set_output_header (bool h) noexcept
	Output a hedaer for the treebank result file.
bool	has_feature (const treebank_feature &fs) const noexcept
	Is a given feature to be calculated?

Protected Attributes
std::array< bool, __treebank_feature_size >	m_what_fs
	The list of features to be computed.
bool	m_check_before_process = true
	Process the treebank file or collection prior to processing.
char	m_separator = '\t'
	Character used as separator.
bool	m_output_header = true
	Output a header for each file.
int	m_be_verbose = 0
	The verbosity of the processor.

Private Member Functions
treebank_error	join_all_files (const std::string &resname) const noexcept
	Joins all resulting files into a single file.

Private Attributes
std::vector< std::string >	m_all_individual_treebank_names
	The list of names of the treebanks.
bool	m_join_files = true
	Join the files into a single file.
size_t	m_num_threads = 1
	Number of threads to use.
std::vector< std::tuple< treebank_error, std::string, std::string > >	m_errors_from_processing
	Set of errors resulting from processing the treebank collection.
std::string	m_out_dir = "none"
	Output directory.
std::string	m_main_file = "none"
	File containing the list of languages and their treebanks.

Detailed Description

Automatic processing of treebank collections.

This class, the objects of which will be referred to as the "processors", has the goal to ease the processing a whole treebank collection and produce data for a fixed set of features available in the library. See the enumeration lal::io::treebank_feature for details on the features available.

This class is meant to process a treebank collection only. A treebank collection is a set of treebank files, each containing several syntactic dependency trees of sentences. Each file is referred to as a treebank file. Each of these files is referenced within a "main file list", henceforth called the "main file". The main file is a two-column formatted file, the first of which contains a self-descriptive name of the treebank that is indicated in the next column.

For example, the main file of a treebank collection of languages could contain:

arb path/to/file/ar-all.heads2
eus path/to/file/eu-all.heads2
ben path/to/file/bn-all.heads2
...

where the first column contains a string referencing the treebank of a language (in this case, by giving an ISO code of a language), and the second column contains the full path to the file with the syntactic dependency trees.

Every processor must be initialised prior to processing the collection. This is done via method init, which requires the path to the main file and the output directory where the results are going to be stored. It also requires a Boolean value indicating whether all (or none) of the features should be used. Moreover, it also admits an optional parameter indicating the number of threads to be used to parallelise the processing of the files.

When initialised, a processor can be removed or added features: when the number of features to calculate is low, it can be initialised with no features, and then be added some via method add_feature. Conversely, if the number of features is high, but not all features are needed, a processer can be initialised with all features, and then be removed some of them via method remove_feature.

Processing a treebank collection with this class will produce a file for every treebank in the collection. These files can be merged together by indicating so via method set_join_files. A new file will be created, regardless of the number of treebanks in the collection.

Finally, the treebank collection is processed via method process. If all the files produced (one for each treebank) are to be joined in a single file, users can give this new file a name by passing it to method process as a string. Also, users can indicate via an optional Boolean parameter whether the individual files are to be removed or not.

Method process returns a value of the enumeration treebank_error. Further errors can be checked via methods get_num_errors, get_error_type, get_error_treebank_filename, get_error_treebank_name.

The usage of this class is a lot simpler than that of class treebank_collection_reader. For example:

treebank_collection_processor tbproc;
// initialise the processor without features (remember to check for errors)
 // and 4 threads for faster processing.
tbproc.init(main_file, output_dir, 4);
tbproc.add_feature(lal::io::treebank_feature::num_crossings);
tbproc.add_feature(lal::io::treebank_feature::var_num_crossings);
tbproc.process();
// it is advisable to check for errors

Member Function Documentation

add_feature()

void lal::io::_process_treebank_base::add_feature ( const treebank_feature & fs )

inlinenoexceptinherited

Adds a feature to the processor.

Parameters

fs	Feature to be added.

get_error_treebank_filename()

const std::string & lal::io::treebank_collection_processor::get_error_treebank_filename ( size_t i ) const

inlinenoexcept

Get the treebank's file name where the ith error happened.

Parameters

i	The index of the error, an unsigned integer.

Returns

The name of the treebank file where the i-th error happened as a string.

get_error_treebank_name()

const std::string & lal::io::treebank_collection_processor::get_error_treebank_name ( size_t i ) const

inlinenoexcept

Get the treebank's name for where the ith error happened.

Parameters

i	The index of the error, an unsigned integer.

Returns

The name of the treebank where the i-th error happened as a string.

get_error_type()

const treebank_error & lal::io::treebank_collection_processor::get_error_type ( size_t i ) const

inlinenoexcept

Get the ith error.

Parameters

i	The index of the error, an unsigned integer.

Returns

A value of the enumeration lal::io::treebank_error.

has_feature()

bool lal::io::_process_treebank_base::has_feature ( const treebank_feature & fs ) const

inlinenoexceptinherited

Is a given feature to be calculated?

Parameters

fs	The feature being queried.

Returns

True or False depending on whether the feature was added or removed.

init()

treebank_error lal::io::treebank_collection_processor::init ( const std::string & main_file, const std::string & output_directory )

noexcept

Initialise the processor with a new collection.

Parameters

main_file	File listing all the treebanks.
output_directory	Directory where the result files are to be stored.

Returns

The type of the error, if any. The list of errors that this method can return is:

• lal::io::treebank_error_type::main_file_does_not_exist
• lal::io::treebank_error_type::output_directory_does_not_exist

join_all_files()

treebank_error lal::io::treebank_collection_processor::join_all_files ( const std::string & resname ) const

privatenoexcept

Joins all resulting files into a single file.

Parameters

resname

Name of the result file.

Returns

An error, if any.

process()

treebank_error lal::io::treebank_collection_processor::process ( const std::string & result_filename = "" )

noexcept

Process the treebank collection.

This method produces the information as explained in this class' description. However, it may fail to do so. In this case it will return a value different from lal::io::treebank_error_type::no_error.

This function uses attributes m_separator, m_output_header to format the output data. It also outputs the current progress if m_be_verbose is set to true.

Moreover, it gathers the errors thay may have occurred during processing. If so, see methods get_num_errors, get_error_type, get_error_treebank_name.

Parameters

result_filename

Name of the file where all values are going to be stored.

Returns

The type of the error, if any. The list of errors that this method can return is:

• lal::io::treebank_error_type::no_features
• lal::io::treebank_error_type::main_file_could_not_be_opened
• lal::io::treebank_error_type::some_treebank_file_failed
• lal::io::treebank_error_type::output_join_file_could_not_be_opened
• lal::io::treebank_error_type::treebank_result_file_could_not_be_opened

See methods get_num_errors, get_error_treebank_filename, get_error_treebank_name to know how to retrieve these errors.

Precondition

Initialisation did not return any errors.

remove_feature()

void lal::io::_process_treebank_base::remove_feature ( const treebank_feature & fs )

inlinenoexceptinherited

Removes a feature from the processor.

Parameters

fs	Feature to be removed.

set_join_files()

void lal::io::treebank_collection_processor::set_join_files ( bool v )

inlinenoexcept

Join the resulting files into a single file.

Parameters

v	A Boolean value.

set_output_header()

void lal::io::_process_treebank_base::set_output_header ( bool h )

inlinenoexceptinherited

Output a hedaer for the treebank result file.

Default is true.

Parameters

h	Output header or not.

set_separator()

void lal::io::_process_treebank_base::set_separator ( char c )

inlinenoexceptinherited

Sets the separator character.

The default seprator is a tabulator character '\t'.

Parameters

c	The separator character.

set_verbosity()

void lal::io::_process_treebank_base::set_verbosity ( int k )

inlinenoexceptinherited

Sets the level of verbosity of the process methods.

Default is 0 (i.e., no verbosity at all). Verbosity is classified by levels:

• Level 1: outputs progress messages.
• Level 2: outputs error messages.

  Parameters

  k	Verbosity level.

Member Data Documentation

m_be_verbose

int lal::io::_process_treebank_base::m_be_verbose = 0

protectedinherited

The verbosity of the processor.

When set to a value greater than or equal to 1, method process will output progress messages.

---

The documentation for this class was generated from the following file:

• lal/io/treebank_collection_processor.hpp