mathdatech/awesome-python
1
0
Fork 0
mirror of https://github.com/vinta/awesome-python.git synced 2024-11-27 22:11:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add Ray to Cluster computing

Browse Source 
# What is this Python project?
Ray is a flexible, high-performance distributed execution framework. It achieves parallelism in Python with simple and consistent API.
Ray is particularly suited for machine learning and forms the base of libraries for deep and reinforcement learning, distributing processing of Pandas dataframes, or hyper parameter search.

# What's the difference between this Python project and similar ones?
 - Similar to Dask, see a comparison here:  https://github.com/ray-project/ray/issues/642
 - Allows to efficiently share large numpy arrays (or objects serializable with Arrow) between the processes, without copying the data and with only minimal deserialization
 - Achieves lower latency with bottom up scheduling
This commit is contained in:
Jan Margeta 2018-10-22 22:22:25 +02:00 committed by GitHub
parent f24b49de18
commit 39fe666fc7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
README.md
View File

@ -248,6 +248,7 @@ Inspired by [awesome-php](https://github.com/ziadoz/awesome-php).
* [faust](https://github.com/robinhood/faust) - A stream processing library, porting the ideas from [Kafka Streams](https://kafka.apache.org/documentation/streams/) to Python. * [faust](https://github.com/robinhood/faust) - A stream processing library, porting the ideas from [Kafka Streams](https://kafka.apache.org/documentation/streams/) to Python.
* [luigi](https://github.com/spotify/luigi) - A module that helps you build complex pipelines of batch jobs. * [luigi](https://github.com/spotify/luigi) - A module that helps you build complex pipelines of batch jobs.
* [mrjob](https://github.com/Yelp/mrjob) - Run MapReduce jobs on Hadoop or Amazon Web Services. * [mrjob](https://github.com/Yelp/mrjob) - Run MapReduce jobs on Hadoop or Amazon Web Services.
* [Ray](https://github.com/ray-project/ray/) - A system for parallel and distributed Python that unifies the machine learning ecosystem.
* [streamparse](https://github.com/Parsely/streamparse) - Run Python code against real-time streams of data via [Apache Storm](http://storm.apache.org/). * [streamparse](https://github.com/Parsely/streamparse) - Run Python code against real-time streams of data via [Apache Storm](http://storm.apache.org/).
## Code Analysis ## Code Analysis