Bring in rework of Section 3

README.md

@@ -1,14 +1,13 @@
 [![DOI](https://zenodo.org/badge/257930838.svg)](https://zenodo.org/badge/latestdoi/257930838)
 
-**Note that the lesson material can change at any point - if you are planning a workshop using this material, 
-either let the maintainers know or make sure you use your own fork of the lesson.**
+## Intermediate Research Software Development Skills In Python Lesson
 
-# Intermediate Research Software Development Skills In Python
-
-An intermediate-level course in research software engineering and development skills and working as part of a 
-team (using Python as an example language). This lesson teaches intermediate-level software development 
-skills in a way that mimics a typical software development 
-process in a team, starting from an [existing piece of software](https://github.com/carpentries-incubator/python-intermediate-inflammation).
+This is an intermediate-level course in collaborative research software engineering and development skills, 
+using Python as an example language. 
+It teaches these skills in a way that mimics a typical software development 
+process working as a part of a team, 
+starting from an [existing piece of software](https://github.com/carpentries-incubator/python-intermediate-inflammation).
+The lesson is developed using [The Carpentries](https://carpentries.org) Jekyll lesson template.
 
 A typical learner for this course may be someone who has gained basic software development skills either by 
 self-learning or attending a foundational course such as the novice [Software Carpentry Python course][swc-lessons]. 
@@ -17,25 +16,33 @@ development-related projects are now becoming larger and more complex and they n
 intermediate software engineering skills to help them design more robust software code, 
 automate the process of testing and verifying its correctness and support collaborations with others.
 
+> :warning: The course material can change at any point - if you are planning a workshop using this material, 
+either let the maintainers know or make sure you use your own fork of the lesson.
+
+The lesson uses [patient inflammation data](https://swcarpentry.github.io/python-novice-inflammation/#scenario-a-miracle-arthritis-inflammation-cure) for code examples, 
+from the [Software Carpentry Python "inflammation" lesson](https://swcarpentry.github.io/python-novice-inflammation/).
+
+Check out the [variant of this lesson](https://github.com/carpentries-incubator/python-intermediate-development-earth-sciences/tree/gh-pages)
+that uses river catchment data in code examples (more suited for Earth and environmental scientists).
+
 ### Lesson Status
 
-The lesson is currently in beta - it has been run over 10 times times with 
+The course is in a stable beta - it has been run over 15 times times with 
 different cohorts by the lesson authors as well as independently by people not directly involved in the lesson development
 and is in a good state to be reused and taught by others.
 
 ## Teaching the Lesson
 
-The lesson is suitable for both instructor-led teaching or guided self-learning where instructors provide help 
+The lesson is suitable for both instructor-led teaching or guided self-learning where helpers provide help 
 and answer questions (synchronously or asynchrounously) as learners go through the course on their own. 
 Initially, in sections 1-3 of the lesson, 
 learners are working on a software project and going though exercises individually.
 In sections 4 and 5, they are grouped and work in teams,
 as they would when collaborating on a team software project development.
 
 The lesson has 5 sections; 
-each section can be delivered or worked through in self-learning mode over a half a day to a day, 
+each section can be delivered in one day by an instructor or worked through in self-learning mode over a half a day to a day, 
 depending on the pace.
-We would recommend setting aside one day per section to make sure the delivery is not rushed. 
 
 If you would like to teach this lesson to your audience and help with more beta testing, 
 please let the lesson developers know by opening an [issue](https://github.com/carpentries-incubator/python-intermediate-development/issues/new?assignees=&labels=pilot&template=lesson-pilot-issue-template.md&title=) with your workshop details and a 

_config.yml

@@ -94,6 +94,11 @@ extras_order:
   - common-issues
   - discuss
   - vscode
+  - software-architecture-extra
+  - programming-paradigms
+  - procedural-programming
+  - functional-programming
+  - object-oriented-programming
   - persistence
   - databases
   - quiz

_episodes/00-setting-the-scene.md

@@ -57,7 +57,27 @@ are transferable to other similar tools and programming languages.
 
 The course is organised into the following sections:
 
+{% comment %}
 ![Course overview diagram](../fig/course-overview.png){: .image-with-shadow width="800px" }
+{% endcomment %}
+
+![Course overview diagram](../fig/course-overview.svg){: .image-with-shadow width="1000px" }
+{% comment %}
+flowchart LR
+A(1. Setting up
+software environment)
+--> B(2. Verifying
+software correctness)
+--> C(3. Software development
+as a process)
+--> D(4. Collaborative
+development for reuse)
+--> E(5. Managing software
+over its lifetime)
+
+https://mermaid.live/edit#pako:eNpdkE1rwzAMhv-K8CmFNrCvSw6D9eO2XlbYYeSiJXJqcKwgKyml9L_PaZox5pOQn8ey3oupuCZTGOv5VB1RFN4_ygDpvGUPORxI1YUG-m5qRrZ6QiGgMDjh0FLQBZRhtXqFdfaYwyeJs-ek_OMrFqFKA8U485vsKQ2YgZoG8tyND8LkYgSETrganUnZZs85bNh7_GZBdQPd2b-2ZQGhPtJd2mUvOewxYDNu8vujSeSBBJxG8M6SupYWZmlakhZdnWK5jFRp9EgtlaZIZU0We6-lKcM1odgrH86hMoVKT0vTdzUqbR02gq0pLPqYulQ7ZdlPUd8Sn8nd7Wa2OwxfzLN3_QE4H4oo
+
+{% endcomment %}
 
 ### [Section 1: Setting up Software Environment](../10-section1-intro/index.html)
 In the first section we are going to set up our working environment

_episodes/10-section1-intro.md

@@ -30,7 +30,32 @@ As you get more comfortable with different tools and their alternatives,
 you will select the one that is right for you based on your personal preferences
 or based on what your collaborators are using.
 
+{% comment %}
 ![Tools needed to collaborate on code development effectively](../fig/section1-overview.png){: .image-with-shadow width="800px" }
+{% endcomment %}
+
+![Tools needed to collaborate on code development effectively](../fig/section1-overview.svg){: .image-with-shadow width="1000px" }
+
+{% comment %}
+flowchart LR
+A(1. Setting up
+software environment
+
+    - Isolate and run code: command line, virtual environment & IDE
+    - Version control and share code: Git & GitHub
+    - Write well-written code: PEP8)
+--> B(2. Verifying
+software correctness)
+--> C(3. Software development
+as a process)
+--> D(4. Collaborative
+development for reuse)
+--> E(5. Managing software
+over its lifetime)
+
+https://mermaid.live/edit#pako:eNpdkttKAzEQhl9lyIVsoS14AtkLQW3RggVRUJC9GXdn20A2s0wmLSK-u0m3K2IuQpj83z-H5MvU3JApTet4X29RFB6fKw9p3RSnc3ghVes3EHsYooFb3aMQkN9ZYd-R18oPdzNYBXaoBOgbkOghm5dp77occdbTFBKmEd1fgwAnsFosw2jzShIsZ9yrsDvYhW3OOhjeW01E2h_ix8i8iU2J9-TcbJ-OSmP2p-XT1aTys9k13BZn82xu28_U1L-GahahWj2FMIFBf1ecpxGMgoZ25LjPBR-HgQEQeuE6MwOyKC7mcMfO4QcLqt3RUfuXbllAKAY6Qsvicg5r9LjJs_6taAB5RwI2zcjZltR2NDFT05F0aJv0cF9ZVRndUkeVKdOxoRaj08pU_jtJMSq_fPralCqRpib2TXqihcWNYGfKFl1IUWqssqyHz3D4E6NyebgZ6R79O_PIff8AoRHADA
+
+{% endcomment %}
 
 Here is an overview of the tools we will be using.
 

_episodes/11-software-project.md

@@ -102,7 +102,7 @@ and then obtain a local copy of that project (from your GitHub) on your machine.
 > from your GitHub account into the home directory on your computer using SSH.
 > Which command(s) would you use to get a detailed list of contents of the directory you have just cloned?
 > 
-> Time: 5 min
+> 
 > > ## Solution
 > > 1. Find the SSH URL of the software project repository to clone from your GitHub account.
 > > Make sure you do not clone the original repository but rather your own fork,
@@ -138,7 +138,7 @@ and then obtain a local copy of that project (from your GitHub) on your machine.
 > {: .solution}
 {: .challenge}
 
-### Our Software Project Structure
+## Our Software Project's Structure
 Let’s inspect the content of the software project from the command line.
 From the root directory of the project,
 you can use the command `ls -l` to get a more detailed list of the contents.
@@ -210,7 +210,7 @@ each of the inflammation data files contains separate trial data for 60 patients
 > Which command(s) would you use to list the contents or a first few 
 > lines of `data/inflammation-01.csv` file?
 > 
-> Time: 1 min
+> 
 > > ## Solution
 > > 1. To list the entire content of a file from the project root do: `cat data/inflammation-01.csv`.
 > > 2. To list the first 5 lines of a file from the project root do: `head -n 5 data/inflammation-01.csv`.
@@ -237,20 +237,22 @@ total 16
 ~~~
 {: .language-bash}
 
-An important thing to note here is that the structure of the project is not arbitrary.
+An important thing to note here is that the structure of our project is not arbitrary.
 One of the big differences between novice and intermediate software development is
 planning the structure of your code.
 This structure includes software components and behavioural interactions between them, 
 including how these components are laid out in a directory and file structure.
 A novice will often make up the structure of their code as they go along.
 However, for more advanced software development,
-we need to plan this structure - called a *software architecture* - beforehand.
+we need to plan and design this structure - called a *software architecture* - beforehand.
 
-Let's have a more detailed look into what a software architecture is
+Let's have a quick look into what a software architecture is
 and which architecture is used by our software project
 before we start adding more code to it.
 
-## Software Architecture
+
+### Software Architecture
+
 A software architecture is the fundamental structure of a software system
 that is decided at the beginning of project development
 based on its requirements and cannot be changed that easily once implemented.
@@ -283,122 +285,28 @@ Another example of modules are classes in object-oriented programming languages.
 >
 {: .callout}
 
-There are various software architectures around defining different ways of
-dividing the code into smaller modules with well defined roles, for example:
-
-- [Model–View–Controller (MVC) architecture](https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller),
-  which we will look into in detail and use for our software project,
-- [Service-oriented architecture (SOA)](https://en.wikipedia.org/wiki/Service-oriented_architecture),
-  which separates code into distinct services,
-  accessible over a network by consumers (users or other services)
-  that communicate with each other by passing data in a well-defined, shared format (protocol),
-- [Client-server architecture](https://en.wikipedia.org/wiki/Client%E2%80%93server_model),
-  where clients request content or service from a server,
-  initiating communication sessions with servers,
-  which await incoming requests (e.g. email, network printing, the Internet),
-- [Multilayer architecture](https://en.wikipedia.org/wiki/Multitier_architecture),
-  is a type of architecture in which presentation,
-  application processing
-  and data management functions
-  are split into distinct layers and may even be physically separated to run on separate machines -
-  some more detail on this later in the course.
-
-### Model-View-Controller (MVC) Architecture
-MVC architecture divides the related program logic
-into three interconnected modules:
-
-- **Model** (data)
-- **View** (client interface),  and
-- **Controller** (processes that handle input/output and manipulate the data).
-
-**Model** represents the data used by a program and also contains operations/rules
-for manipulating and changing the data in the model.
-This may be a database, a file, a single data object or a series of objects -
-for example a table representing patients' data.
-
-**View** is the means of displaying data to users/clients within an application
-(i.e. provides visualisation of the state of the model).
-For example, displaying a window with input fields and buttons (Graphical User Interface, GUI)
-or textual options within a command line (Command Line Interface, CLI) are examples of Views.
-They include anything that the user can see from the application.
-While building GUIs is not the topic of this course,
-we will cover building CLIs in Python in later episodes.
-
-**Controller** manipulates both the **Model** and the **View**.
-It accepts input from the **View**
-and performs the corresponding action on the **Model** (changing the state of the model)
-and then updates the **View** accordingly.
-For example, on user request,
-**Controller** updates a picture on a user's GitHub profile
-and then modifies the **View** by displaying the updated profile back to the user.
-
-#### MVC Examples
-
-MVC architecture can be applied in scientific applications in the following manner.
-Model comprises those parts of the application that deal with
-some type of scientific processing or manipulation of the data,
-e.g. numerical algorithm, simulation, DNA.
-View is a visualisation, or format, of the output,
-e.g. graphical plot, diagram, chart, data table, file.
-Controller is the part that ties the scientific processing and output parts together,
-mediating input and passing it to the model or view,
-e.g. command line options, mouse clicks, input files.
-For example, the diagram below depicts the use of MVC architecture for the
-[DNA Guide Graphical User Interface application](https://www.software.ac.uk/developing-scientific-applications-using-model-view-controller-approach).
-
-![MVC example of a DNA Guide Graphical User Interface application](../fig/mvc-DNA-guide-GUI.png){: .image-with-shadow width="400px" }
-{% comment %}Image from https://www.software.ac.uk/developing-scientific-applications-using-model-view-controller-approach{% endcomment %}
-
-> ## Exercise: MVC Application Examples From your Work
-> Think of some other examples from your work or life
-> where MVC architecture may be suitable
-> or have a discussion with your fellow learners.
->
-> Time: 5 min
-> > ## Solution
-> > MVC architecture is a popular choice when designing web and mobile applications.
-> > Users interact with a web/mobile application by sending various requests to it.
-> > Forms to collect users inputs/requests
-> > together with the info returned and displayed to the user as a result represent the View.
-> > Requests are processed by the Controller,
-> > which interacts with the Model to retrieve or update the underlying data.
-> > For example, a user may request to view its profile.
-> > The Controller retrieves the account information for the user from the Model
-> > and passes it to the View for rendering.
-> > The user may further interact with the application
-> > by asking it to update its personal information.
-> > Controller verifies the correctness of the information
-> > (e.g. the password satisfies certain criteria,
-> > postal address and phone number are in the correct format, etc.)
-> > and passes it to the Model for permanent storage.
-> > The View is then updated accordingly and the user sees its updated profile details.
-> >
-> > Note that not everything fits into the MVC architecture
-> > but it is still good to think about how things could be split into smaller units.
-> > For a few more examples, have a look at this short
-> > [article on MVC from CodeAcademy](https://www.codecademy.com/articles/mvc).
-> {: .solution}
-{: .challenge}
+We are going to talk about software architecture and design a 
+bit more in [Section 3](../30-section3.html) - for now
+it is sufficient to know that the way our software project's code is structured is intentional.
 
-> ## Separation of Concerns
-> Separation of concerns is important when designing software architectures
-> in order to reduce the code's complexity.
-> Note, however, there are limits to everything -
-> and MVC architecture is no exception.
-> Controller often transcends into Model and View
-> and a clear separation is sometimes difficult to maintain.
-> For example, the Command Line Interface provides both the View
-> (what user sees and how they interact with the command line)
-> and the Controller (invoking of a command) aspects of a CLI application.
-> In Web applications, Controller often manipulates the data (received from the Model)
-> before displaying it to the user or passing it from the user to the Model.
->
-{: .callout}
 
 ### Our Project's Architecture
 
-Our software project uses the MVC architecture.
-The file `inflammation-analysis.py` is the **Controller** module
+Our software project uses the [Model-View-Controller (MVC) architecture](https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller). 
+MVC architecture divides the software logic into three interconnected modules:
+
+- **Model** (data) - represents the data used by a program and contains operations/rules
+  for manipulating and changing the data in the model (a database, a file, a single data object 
+  or a series of objects - for example a table representing patients' data).
+- **View** (client interface) - provides means of displaying data to users/clients within an 
+  application (i.e. provides visualisation of the state of the model).
+  For example, displaying a window with input fields and buttons (Graphical User Interface, GUI)
+  or textual options within a command line (Command Line Interface, CLI) are examples of Views.
+- **Controller** (processes that handle input/output and manipulate the data) - 
+  accepts input from the **View** and performs the corresponding action on the **Model** 
+  (changing the state of the model) and then updates the **View** accordingly.
+
+In our project, `inflammation-analysis.py` is the **Controller** module
 that performs basic statistical analysis over patient data
 and provides the main entry point into the application.
 The **View** and **Model** modules are contained in the files `views.py` and `models.py`, respectively,
@@ -407,7 +315,7 @@ Data underlying the **Model** is contained within the directory `data` -
 as we have seen already it contains several files with patients’ daily inflammation information.
 
 We will revisit the software architecture and MVC topics once again in later episodes
-when we talk in more detail about [software design](../32-software-design/index.html).
+when we talk in more detail about [software architecture and design](../32-software-architecture-design/index.html).
 We now proceed to set up our virtual development environment
 and start working with the code using a more convenient graphical tool -
 [IDE PyCharm](https://www.jetbrains.com/pycharm/).

_episodes/12-virtual-environments.md

@@ -63,10 +63,11 @@ which contains a particular version of Python installation
 plus a number of additional external libraries.
 
 Virtual environments are not just a feature of Python -
-most modern programming languages use them to isolate libraries for a specific project
-and make it easier to develop, run, test and share code with others.
-Even languages that don't explicitly have virtual environments have other mechanisms
-that promote per-project library collections.
+most modern programming languages use a similar mechanism to isolate libraries or dependencies
+for a specific project, making it easier to develop, run, test and share code with others.
+Some examples include Bundler for Ruby, Conan for C++, or Maven with classpath for Java.
+This can also be achieved with more generic package managers like Spack,
+which is used extensively in HPC settings to resolve complex dependencies.
 In this episode, we learn how to set up a virtual environment to develop our code
 and manage our external dependencies.