How to Deploy Enterprise-Grade Virtual Python Environments Using Pyramid
The article explains how Pyramid’s Virtual Environments manager addresses enterprise challenges in deploying Python by enabling administrators to create, manage, and secure multiple coexisting Python virtual environments with different versions across multi-node, load-balanced AI servers, facilitating scalable, multi-tenant, and multi-project Python processing exemplified by a data scientist using it for complex ETL and analytics on SAP Hana data.
Python is one of the world’s most popular programming languages, used extensively for machine learning and AI projects, with many open-source, free programs and projects added daily. However, when deploying Python in an enterprise, multi-user, multi-project environment, there are significant challenges:
- Lack of proper, functional versioning: Python does not have inherent versioning switches and can only deploy specific versions of third-party packages for a given installation. This makes it difficult to run different Python projects using a single setup, as each may require different versions of the language engine or packages.
- No out-of-the-box mechanism for load-balanced, multi-node processing: Python does not natively support distributing workloads across multiple servers.
Most third-party BI tools treat Python as a light add-on, exposing limited capabilities within single-version environments.
Pyramid’s Solution
Pyramid’s Virtual Environments manager allows administrators to create and govern multiple Python environments on one or more dedicated “AI” servers. Key features include:
- Ability to create, add, remove, and edit multiple virtual environments on one or more servers.
- Support for different versions of Python operating on the same host, enabling multiple coexisting environments.
- Security and exposure controls for different tenants in a multi-tenant deployment.
Multi-node Processing
Each virtual environment can be hosted on one or more servers, allowing Pyramid to deliver a multi-node, load-balanced solution for Python processing. Customers can replicate virtual environments across the Pyramid cluster for scalability.
Business Case Example
Ingrid, a data scientist at GIA Logistics, uses Pyramid to create analytical reports and dashboards on operational data stored in an SAP Hana database. She needs to:
- Run a Python script that reads an input table from SAP Hana, combines multiple CSV files, and performs ETL data cleaning using the “geopy” library version 1.17.1 on Python 3.6.3.
- Use another script that calculates the geo distance from the closest branch, requiring “geopy” version 2.0.0 on Python 3.7 due to functionality limitations in the earlier package version.
Both scripts must be included in her data model, even though they use different versions of the same library and different Python versions. Multiple workflows and servers are required to accommodate the workload.
GIA Logistics deploys two Pyramid AI servers with both Python 3.6.3 and 3.7 environments to support multiple users running Python scripts.
- Ingrid copies her first script (using the Python Legacy Environment with geopy 1.7.1 on Python 3.6.3) to retrieve address coordinates and perform ETL/data cleaning.
- She then copies her second script (using the Pyramid Python Environment with geopy 2.0.0 on Python 3.7) to calculate geo distances.
- Ingrid can print a report with all latitude and longitude details, which are now stored in the SAP Hana database for reuse.
Summary
Python is widely used for machine learning and AI but lacks robust versioning and load balancing. Most BI tools offer only single-version Python environments, limiting their use to small data sets and business problems.
Pyramid addresses these challenges by allowing administrators to deploy multiple coexisting virtual Python environments on multiple servers. This enables the installation of any package version, supports multiple users and projects, and provides an enterprise-grade approach to Python-based analytics.