First-class Python integration in Pyramid - Pyramid Analytics

Python has rapidly become developers’ top choice for machine learning and deep learning projects. Machine learning allows users to feed massive amounts of data into a computer algorithm to get data-driven recommendations and decisions based only on input data. Pyramid enables Python code to be used either as a data source or for data manipulations on up to billions of records, allowing resultant data to be written back to a database. The integration of Python into Pyramid’s self-service BI, with its powerful drag-and-drop data extraction and preparation tools, provides non-technical analysts with the ability to work extensively with Python functions without having to write any code.

The problem

The Python language is an overwhelming favorite for machine learning and development in general. However, users wanting to perform analytics still require a self-service BI tool to create dashboards, presentations, and reports using the output from Python. Reuse of code remains a huge challenge for enterprise users who need to share code as well as access data sources, dashboards, reports, and publications. Python’s siloed environment excludes non-coders from testing and further analyzing models using self-service BI tools.

Third-party vendors like Power BI and Tableau have serious limitations in integrating Python. With those tools, Python code is not run as a batch processing job—instead, it is primarily run inline when generating discoveries, so it’s always limited to small data sets and small business problems, not real-world machine learning challenges. In these scenarios, the Python scripts are processed against aggregated values rather than on the granular data. The other major gap is that Python ML models trained on a data set cannot be easily created, saved, and shared—let alone reused—to predict values on the larger population data set.

In many respects, most BI tools treat Python as a side-show gimmick without any serious treatments or integrations.

Pyramid’s solution

In stark contrast, Pyramid has made Python (and R) a first-class citizen in its architecture and product strategy.

Pyramid integrates Python into its Model tool, allowing users to utilize Python code as a data source or as a data manipulation and calculation element. The Model tool allows Python to perform basic aggregation, calculations, and data manipulations on data using any degree of logic afforded by Python. Python machine learning models can also be trained, tested, and then deployed to run predictions on millions or billions of records and can be shared and re-used in other data sources.

Governance of the Python scripts is also maintained with managed access and content versioning. The process of extracting and preparing data for a learning script that is normally performed by a Python developer can be performed using drag-and-drop actions by other professionals without deep Python skills, saving huge development effort. The ability to write back results from the Python code to the original data source provides a powerful means of adding to the value chain built by Pyramid.

A marketplace with free reusable Python source code provides non-technical analysts with a large library of predefined functions. Further editing, code-creation, testing, and running of Python code can all be performed from within Pyramid. Pyramid’s entire self-service BI is then available for all analysts to create dashboards, reports, publications, and more from the resultant data model.

In this blog, the combination of Python and Pyramid is illustrated in the context of stock market trading, where an analyst wants to quickly and easily source stock market data and create visualizations using Pyramid’s powerful self-service BI tools. In future blogs, further use cases such as forecasting and learn-and-predict scenarios will be demonstrated.

Stock market example

Using the Model tool in Pyramid with Python as a source, two separate Python scripts from the IEXFinance service are used for company data and stock price data. The first script fetches close, high, low, and open prices, and volumes.

The second IEXFinance script retrieves the company name, exchange, industry, sector, etc. Both scripts are included in the Model tool that performs the ETL. Pyramid’s time intelligence is applied, and the data is stored in Pyramid’s in-memory database (but could be stored in any of the 27 target databases/repositories supported by Pyramid).

By simply dragging and dropping month, company, and close price into the drop zones—and then adding a year filter—an interactive report can be created that can be further sliced and diced according to all hierarchies and measures that are available.

To create a candlestick chart using open, close, high, and low prices, and then further analyze per company by creating an additional company filter, the measures are dropped in the pertinent pre-defined areas for the candlestick chart, defined in the “Additional Values” area in the drop zone, instantly providing a clear picture of a selected company’s performance over time.

To further enhance the graphic depiction, the difference in the daily open and close price can be displayed with just a few points and clicks.

To do this, the Create Measure function (Constructed Functions, Subtraction) is used to add a new “Difference” measure to the “Color” dropdown, selecting positive and negative as a method of display. Then the volume measure is added, selecting secondary spline area chart as the display method, resulting in a graphically pleasing, intuitive chart that can be further manipulated at will. All of this can be done without programming or “heavy lifting.”

Using the Model tool’s scheduler, a schedule can be created (using any time interval from minutes to months) to run the entire data flow and reports in an automation.

Summary

Pyramid enables Python code to be used either as a data source or for data manipulations, allowing resultant data to be written back to a database. Python users wanting to perform analytics require a self-service BI tool to access data sources and create dashboards, presentations, and reports. The Python environment excludes non-coders from testing—and further, analyzing models using self-service BI tools. Third party vendors have serious limitations integrating Python: it is limited to small data sets, can only be performed on the aggregated values, and cannot be performed on values created using Python. Python Machine models cannot be created, saved, and shared.

In contrast, Pyramid integrates Python into its Model tool, allowing users to utilize Python code as a data source or as a data manipulation and calculation add-on to existing data sources. Machine learning scripts can be shared and re-used in other data sources and the governance of the Python scripts is maintained with managed access and content versioning.

In addition, non-technical team members can extract and prepare data using drag and drop functionality, saving huge development effort. Pyramid provides the ability to write back results from the Python code to the original data source. A marketplace with free reusable Python source code provides non-technical analysts with a large library of predefined functions. Editing, code-creation, testing, and running of Python code can all be performed from within Pyramid. Pyramid’s entire self-service BI is then available for all analysts to create dashboards, reports, publications, etc. from the resultant data model.