-
Notifications
You must be signed in to change notification settings - Fork 1
Refactor #2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Refactor #2
Conversation
LLParse
commented
Jul 15, 2015
- Rework configuration system to use typesafe snakeyaml api
- Minimize dependency scope, remove unused dependencies for ~50% size savings in built artifacts
- Define protocol buffer messages for journal, name, data nodes
- Remove configuration logic from artifact server, use Mesos data fields to pass protobuf between scheduler & executor (this fixes idempotency [WIP, name/data nodes not quite finished])
- Refactor/simplify executor system, emit hdfs-site.xml using {{mustache}} template and task protobuf data
- Pool enough offers to launch journalnodes all at once (TODO: save journal rpc addresses in persistent state, use ports in offer when configured ports are set to '0', genericize pooled offer logic to work for name/data node acquisition phases, wrap offer system in timer to prevent holding onto resources indefinitely in too-few-resources scenarios)
- TODO: major scheduler refactor, tweak name/datanode config
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of using a yaml config here, what I would prefer to do is something similar to the following PR (which didn't get finished), mesosphere-backup#123. I would like to use environment variables or command line arguments (similar to Cassandra). These values could then be used to generate the *-site.xml configs used by HDFS.
|
Overall comment, this would need to be rebased with mesosphere/master |