Restructure of software development as a process section

[thomaskileyukaea]

Restructure the software development process to instead demonstrate refactoring existing, poorly architect-ed code into better code.

Instead of directly teaching functional, OOP etc it would show example code that could be improved (made more readable, testable, extendable) by applying these techniques. It would then name them, so that people can read more.

By focusing on examples where each helps it grounds why you would apply them, rather than potentially just encouraging people to think they need to use classes to being doing "good coding".

Another motivation for this change is we got some feedback when running the course is mostly what people are doing is refactoring legacy projects rather than writing new code and this would develop the skills for doing that.

Goals of this rework:

• Practical experience of refactoring code rather than writing fresh code, mirroring what most people will experience
• Understand the value of OOP and functional techniques and when they are appropriate to apply
• Easier/faster to teach as less theoretical information, more examples to work through
• Encourage a YAGNI approach to good code structure

Proposed Structure

1. <Potentially keep / drop the requirements section>
2. Clean code
   2.1. Introduce what an abstraction is
   2.2. Introduce what refactoring is
3. Refactoring functions to just do one thing
   3.1. Trying to test a messy function that both reads from a path and does some calculation
   3.2. Refactor into a pure function that does the calculation using the output of another function that reads the file
   3.3. Writing tests for the new pure function
4. Refactoring how we display the results from computing the result
   4.1 Refactor out logic for computing something from displaying it in a graph
   4.2 Add a new view
   4.3 Introduce MVC and patterns as a source for inspiration
5. Adding persistence
   5.1. Ideas of an interface in programming concept
   5.2. Write an interface for writing outputs
   5.3. Writing an implementation of the interface
   5.4. Using the interface
   5.4. Writing an alternative implementation.
   5.5. Explain the concept of de-coupling

Depending on how much quicker this section is to teach I think the following exercises would complement it well:

• Architecting a large project using boxes and lines (probably between 4 and 5, or even make it for 5)
• Refactoring something into layers (probably after or worked into 4)
• Using encapsulation to keep data in a consistent state (after 5)

[bielsnohr]

I fully endorse the stated goals and justification to root this section more in refactoring rather than writing fresh code. As mentioned in more than a few meetings, this section is always the most difficult to deliver, and I think having a different approach will give us some insight into what works best for this topic. I think there is also pretty good evidence for this from a similar course being developed: https://third-bit.com/sdxjs/

A few comments on the proposed structure.

Proposed Structure

1. <Potentially keep / drop the requirements section>

I think we will need to keep this in some form because feedback from AstraZeneca was that this is an important part of their projects, and they wanted something about it in the course. I tend to agree that it needs to be somewhere. Whether it needs to be here, that is a different question. I could see it being moved to Section 5 since requirements collection does fit more into the project management side of the software development lifecycle. We would need to see what the implications of that are for Section 4 if it refers back to the requirements at any point.

2. Clean code
   2.1. Introduce what an abstraction is
   2.2. Introduce what refactoring is

I'm not a huge fan of the name "Clean Code" because of association with Rob Martin ("Uncle Bob")---discussion for another time. But it does describe what the objective is, and I don't have a good alternative name.

3. Refactoring functions to just do one thing
   3.1. Trying to test a messy function that both reads from a path and does some calculation
   3.2. Refactor into a pure function that does the calculation using the output of another function that reads the file
   3.3. Writing tests for the new pure function

Really like the sound of this. ^

4. Refactoring how we display the results from computing the result
   4.1. Refactor out logic for computing something from displaying it in a graph
   4.2. Add a new view
   4.3. Introduce MVC and patterns as a source for inspiration

Great

5. Adding persistence
   5.1. Ideas of an interface in programming concept
   5.2. Write an interface for writing outputs
   5.3. Writing an implementation of the interface
   5.4. Using the interface
   5.4. Writing an alternative implementation.
   5.5. Explain the concept of de-coupling

Interestingly, there was a module on Persistence in the past, that is now in the "Extras". Might be helpful to start from that? https://carpentries-incubator.github.io/python-intermediate-development/persistence/index.html

Depending on how much quicker this section is to teach I think the following exercises would complement it well:

• Architecting a large project using boxes and lines (probably between 4 and 5, or even make it for 5)

Agreed, this would be great to have, and I would give it priority of the three additional exercises mentioned in this section.

• Refactoring something into layers (probably after or worked into 4)
• Using encapsulation to keep data in a consistent state (after 5)

[thomaskileyukaea]

The original questions, points and objectives.
✔️ - this remains in the new version
❌ - this is intentionally removed from the new version
❓ - consider adding to the new version
▶️ - this maps on to a new question

Software design

questions:

• What should we consider when designing software? ✔️
• How can we make sure the components of our software are reusable? ▶️ 3.5: Understand what decoupled code is, and why you would want it.

objectives:

• Understand the use of common design patterns to improve the extensibility, reusability and overall quality of software. ▶️ 3.4: Understand the MVC pattern and how to apply it
• Understand the components of multi-layer software architectures. ❓ Depends on whether we include section on layered architecture

keypoints:

• Planning software projects in advance can save a lot of effort and reduce 'technical debt' later - even a partial plan is better than no plan at all. ❌ (replaced by the idea of YAGNI)
• By breaking down our software into components with a single responsibility, we avoid having to rewrite it all when requirements change ✔️
• When writing software used for research, requirements will almost always change. ✔️
• 'Good code is written so that is readable, understandable, covered by automated tests, not over complicated and does well what is intended to do.' ✔️

Programming Paradigms

questions:

• How does the structure of a problem affect the structure of our code? ❓ (maybe part of separating out considerations)
• How can we use common software paradigms to improve the quality of our software? ❌ (rather than centering software paradigms, we invert this question, how do we improve quality of our software - using applicable paradigms)

objectives:

• Describe some of the major software paradigms we can use to classify programming languages. ❌(this is no longer an objective - the point is to write better code rather than be familiar classification of programming languages).

keypoints:

• A software paradigm describes a way of structuring or reasoning about code. ❌Not covering what a software paradigm is
• Different programming languages are suited to different paradigms. ❌ Not covering what a software paradigm is
• Different paradigms are suited to solving different classes of problems. ▶️ 3.3 Functional programming will talk about when it is and isn't suited to solving a problem
• A single piece of software will often contain instances of multiple paradigms. ❌ Not covering what a software paradigm is

Functional Programming

questions:

• What is functional programming? ▶️ 3.3 Functional programming
• Which situations/problems is functional programming well suited for? ▶️ 3.3 Functional programming

objectives:

• Describe the core concepts that define the functional programming paradigm ❌Not covering what a software paradigm is
• Describe the main characteristics of code that is written in functional programming style ❌Instead focusing on the benefits and when applicable of functional programming
• Learn how to generate and process data collections efficiently using MapReduce and Python's comprehensions ❌Not covering advanced python specific concepts and would instead just link to docs

keypoints:

• Functional programming is a programming paradigm where programs are constructed by applying and composing smaller and simple functions into more complex ones (which describe the flow of data within a program as a sequence of data transformations). ▶️ 3.3 Functional programming
• In functional programming, functions tend to be pure - they do not exhibit side-effects (by not affecting anything other than the value they return or anything outside a function). Functions can also be named, passed as arguments, and returned from other functions, just as any other data type. ❌ Is kind of too specific about functional programming.
• MapReduce is an instance of a data generation and processing approach, in particular suited for functional programming and handling Big Data within parallel and distributed environments. ❌Not covering advanced python specific concepts and would instead just link to docs
• Python provides comprehensions for lists, dictionaries, sets and generators - a concise (if not strictly functional) way to generate new data from existing data collections while performing sophisticated mapping, filtering and conditional logic on original dataset's members. ❌Not covering advanced python specific concepts and would instead just link to docs

Object Oriented Programming

questions:

• How can we use code to describe the structure of data? ❓ Perhaps we should have a section on structuring data - it is important and useful in both OOP and Functional styles, and is a good way to make data more readable. Could also cover things like type safety, xarray
• How should the relationships between structures be described? ❓ see above

objectives:

• Describe the core concepts that define the object oriented paradigm ❌ not specifically defining OOP - instead introducing relevant concepts from it
• Use classes to encapsulate data within a more complex program ❌ rather than just talking about encapsulation, instead introduce it in the context of polymorphism to better shows it function beyond simply having functions that do one thing
• Structure concepts within a program in terms of sets of behaviour ❌ focusing instead of polymorphism
• Identify different types of relationship between concepts within a program ❓ I don't know what this means
• Structure data within a program using these relationships ❓ see above in questions about maybe having a section on structuring data

keypoints:

• Object oriented programming is a programming paradigm based on the concept of classes, which encapsulate data and code. ❌ not specifically defining OOP
• Classes allow us to organise data into distinct concepts. ❌ rather than just talking about encapsulation, instead introduce it in the context of polymorphism
• By breaking down our data into classes, we can reason about the behaviour of parts of our data. ❓ I don't understand this
• Relationships between concepts can be described using inheritance (is a) and composition (has a). ❓ Might be worth talking about this in the inheritance bit

Architecture Revisited: Extending Software

questions:

• How can we extend our software within the constraints of the MVC architecture? ❌ Not sure what this question is for - but new version will demonstrate ways of extending the application into model and view

objectives:

• Extend our software to add a view of a single patient in the study and the software's command line interface to request a specific view. ❌ Instead refactor our software to split up model code from view code

keypoints:

• By breaking down our software into components with a single responsibility, we avoid having to rewrite it all when requirements change. Such components can be as small as a single function, or be a software package in their own right. ✔️

[anenadic]

Will be fixed via #333.