diff --git a/content/pages/01-introduction/01-learning-programming.markdown b/content/pages/01-introduction/01-learning-programming.markdown index 217546782..08f676d8e 100644 --- a/content/pages/01-introduction/01-learning-programming.markdown +++ b/content/pages/01-introduction/01-learning-programming.markdown @@ -125,11 +125,6 @@ repositories and sites with practice problems and solutions: * [TeachCraft](https://teachcraft.net/) combines Minecraft with Python to learn coding. -* [500 Data Structures and Algorithms practice problems and their solutions](https://techiedelight.quora.com/500-Data-Structures-and-Algorithms-practice-problems-and-their-solutions) - covers a large swath of the computer science space. It is not important - to know all of these algorithms and data structures but experience with - many of them will be greatly beneficial in becoming a better developer. - ### First-hand advice These articles are written by programmers who explain how they learned to @@ -145,11 +140,6 @@ give example paths you can think about taking as a beginner: including persistence, respecting others and considering ideas that are outside your comfort zone. -* [Mastering programming](https://www.facebook.com/notes/kent-beck/mastering-programming/1184427814923414) - by [Kent Beck](https://en.wikipedia.org/wiki/Kent_Beck) contains - patterns and observations for how experienced programmers he has worked - with in the past became great software developers. - * [This Picture Will Change the Way You Learn to Code](https://dev.to/nextdotxyz/this-picture-will-change-the-way-you-learn-tocode-4kmh) covers a well done graphics of many up-to-date concepts and tools that developers use. The post reminds you that you will not and should not learn @@ -170,10 +160,6 @@ your teaching experience: is an awesome resource that explains how you can use simple but fun drawings to teach otherwise difficult technical concepts to students. -* [Teaching programming to working professionals](http://pgbovine.net/PG-Podcast-21-Trey-Hunner.htm) - is a video podcast with [Trey Hunter](https://twitter.com/treyhunner) - about his experience teaching Python to experienced professionals. - * [Teach Yourself Computer Science](https://teachyourselfcs.com/) is intended as a self-teaching tool with many resources that are classic computer science textbooks. There are also nice explanations for why diff --git a/content/pages/01-introduction/02-python-programming-language.markdown b/content/pages/01-introduction/02-python-programming-language.markdown index 6cc9be260..a3b046bb1 100644 --- a/content/pages/01-introduction/02-python-programming-language.markdown +++ b/content/pages/01-introduction/02-python-programming-language.markdown @@ -79,9 +79,6 @@ Dictionary comprehension: ### General Python language resources -* The [online Python tutor](http://www.pythontutor.com/) visually walks - through code and shows how it executes on the Python interpreter. - * [Python Module of the Week](http://pymotw.com/2/index.html) is a tour through the Python standard library. @@ -164,10 +161,6 @@ the very beginner topics. covers what the code for list comprehensions looks like and gives some example code to show how they work. -* [An Introduction to Python Lists](http://effbot.org/zone/python-list.htm) - is a solid overview of Python lists in general and tangentially covers - list comprehensions. - ### Python generator resources * This blog post entitled diff --git a/content/pages/01-introduction/03-why-use-python.markdown b/content/pages/01-introduction/03-why-use-python.markdown index 505585cf3..86a0e3ecd 100644 --- a/content/pages/01-introduction/03-why-use-python.markdown +++ b/content/pages/01-introduction/03-why-use-python.markdown @@ -33,10 +33,7 @@ The IEEE ranked Python as the [#1 programming language in 2019](https://spectrum.ieee.org/computing/software/the-top-programming-languages-2019), which continued its hot streak after ranking it -[#1 in 2018](https://spectrum.ieee.org/at-work/innovation/the-2018-top-programming-languages), -[#1 in 2017](https://spectrum.ieee.org/at-work/innovation/the-2017-top-programming-languages) -and -[#3 top programming language in 2016](http://spectrum.ieee.org/computing/software/the-2016-top-programming-languages). +[#1 in 2018](https://spectrum.ieee.org/at-work/innovation/the-2018-top-programming-languages) and 2017. [RedMonk's June 2019 ranking](https://redmonk.com/sogrady/2019/07/18/language-rankings-6-19/) had Python at #3, which held consistent from previous years' rankings in [2018](https://redmonk.com/sogrady/2018/08/10/language-rankings-6-18/) @@ -133,13 +130,6 @@ language. system versus statically typed languages, be sure to [read this thorough explanation of the topic](http://blogs.perl.org/users/ovid/2010/08/what-to-know-before-debating-type-systems.html). -* [Why I swapped C#.NET for Python as my default language and platform (and won’t be going back)](https://medium.com/@anthonypjshaw/why-i-swapped-c-net-for-python-as-my-default-language-and-platform-and-wont-be-going-back-e0063a25e491) - provides a viewpoint from someone who is not a professional developer but - uses coding to hack out some projects. He found Microsoft's .NET ecosystem - lacking when it came to satisfying his needs and Python filled the gap for - him with its wide array of open source code libraries, package management - and ability to work well on platforms other than Windows. - * [Python, Machine Learning, and Language Wars](http://sebastianraschka.com/blog/2015/why-python.html) compares Python with R, MATLAB and Julia for data science work. While Python is great for [deployment automation](/deployment.html) and diff --git a/content/pages/01-introduction/04-python-2-or-3.markdown b/content/pages/01-introduction/04-python-2-or-3.markdown index eaf166982..a362e7b49 100644 --- a/content/pages/01-introduction/04-python-2-or-3.markdown +++ b/content/pages/01-introduction/04-python-2-or-3.markdown @@ -78,7 +78,7 @@ gone through the process and have advice for making it less painful. implementations. There is also a [quick reference for writting code with Python 2 and 3 compatibility](https://wiki.python.org/moin/PortingToPy3k/BilingualQuickRef). -* [Upgrading to Python 3 with Zero Downtime](https://tech.yplanapp.com/2016/08/24/upgrading-to-python-3-with-zero-downtime/) +* [Upgrading to Python 3 with Zero Downtime](https://adamj.eu/tech/2016/08/24/upgrading-yplan-to-python-3-with-zero-downtime/) supplies advice on transitioning a large existing Python 2 web application to Python 3. Their process involved upgrading dependencies, testing and deploying the new version before going back to clean up unnecessary code diff --git a/content/pages/01-introduction/05-enterprise-python.markdown b/content/pages/01-introduction/05-enterprise-python.markdown index f6f1ea7f8..c50052847 100644 --- a/content/pages/01-introduction/05-enterprise-python.markdown +++ b/content/pages/01-introduction/05-enterprise-python.markdown @@ -70,7 +70,7 @@ best maintained and fully featured pieces of code for any language. Meanwhile, some of the traditional enterprise software development languages such as Java have languished due to underinvestment by their major corporate -backers. When [Oracle purchased Sun Microsystems in 2009](http://www.oracle.com/us/corporate/press/018363) +backers. When Oracle purchased Sun Microsystems in 2009, there was a long lag time before Java was enhanced with new language features in Java 7. Oracle also [bundles unwanted adware with the Java installation](http://www.engadget.com/2015/03/06/java-adware-mac/), diff --git a/content/pages/01-introduction/06-community.markdown b/content/pages/01-introduction/06-community.markdown index 1baec5f4f..a5f463319 100644 --- a/content/pages/01-introduction/06-community.markdown +++ b/content/pages/01-introduction/06-community.markdown @@ -77,7 +77,7 @@ resources provide perspective on offline events like and [PyCon US 2016](http://www.dreisbach.us/blog/pycon-2016/). There are many other retrospectives for other - [community-led conferences such as EuroPython](http://www.artima.com/weblogs/viewpost.jsp?thread=261930). + [community-led conferences such as EuroPython](https://www.artima.com/weblogs/viewpost.jsp?thread=261930). These summaries can be a great way to get a slice of the experience before purchasing a ticket and booking a trip. @@ -112,16 +112,11 @@ the community. provides a starter page with links to community-run newsletters, resources and conferences. -* There are many large active online communities on - [Reddit](https://www.reddit.com/r/python) and - [IRC channels](https://freenode.net/) such as #python, #python-dev - and #distutils. - * The Python community has a concept known as "Benevolent Dictator For Life" that may appear odd to newcomers. Essentially, Guido Van Rossum created the language and still has the ability to decide community arguments one way or the other. This post on the - [origin of BDFL](http://www.artima.com/weblogs/viewpost.jsp?thread=235725) + [origin of BDFL](https://www.artima.com/weblogs/viewpost.jsp?thread=235725) has more context about Guido's role. * [Python Community and Python at Dropbox](https://talkpython.fm/episodes/show/30/python-community-and-python-at-dropbox) diff --git a/content/pages/01-introduction/07-companies.markdown b/content/pages/01-introduction/07-companies.markdown index 7eb7a78f2..c4e02cfc8 100644 --- a/content/pages/01-introduction/07-companies.markdown +++ b/content/pages/01-introduction/07-companies.markdown @@ -37,7 +37,7 @@ below). at job descriptions on sites like [Glassdoor with "Python Goldman Sachs" keywords](https://www.glassdoor.com/Jobs/Goldman-Sachs-python-Jobs-EI_IE2800.0,13_KO14,20.htm) and - [Indeed for JP Morgan Chase](https://www.indeed.com/salaries/Python-Developer-Salaries-at-JPMorgan-Chase). + [Indeed for JP Morgan Chase](https://www.indeed.com/cmp/JPMorgan-Chase/salaries/Python-Developer). Salaries and responsibilities vary widely based on the role and whether Python is used for data analysis, [web application development](/web-development.html) or DevOps. diff --git a/content/pages/01-introduction/08-best-python-resources.markdown b/content/pages/01-introduction/08-best-python-resources.markdown index 81113487a..6677ba3d2 100644 --- a/content/pages/01-introduction/08-best-python-resources.markdown +++ b/content/pages/01-introduction/08-best-python-resources.markdown @@ -47,13 +47,6 @@ should skip down to the next section for "experienced developers". and problems rather than jumping into a specific language that's recommended to you by a friend. -* [A Python Crash Course](https://www.grahamwheeler.com/posts/python-crash-course.html) - gives an awesome overview of the history of Python, what drives the - programming community and dives into example code. You will likely need - to read this in combination with other resources to really let the syntax - sink in, but it's a great article to read several times over as you - continue to learn. - * The [Python projects tag](https://www.twilio.com/blog/tag/python) on the Twilio blog is constantly updated with fun tutorials you can build to learn Python, such as the @@ -61,9 +54,6 @@ should skip down to the next section for "experienced developers". [Choose Your Own Adventures Presentations using Flask and WebSockets](https://www.twilio.com/blog/2014/11/choose-your-own-adventure-presentations-with-reveal-js-python-and-websockets.html) and [Martianify Photos with OpenCV](https://www.twilio.com/blog/2015/11/getting-started-with-opencv-and-python-featuring-the-martian-2.html). -* [A Byte of Python](http://www.swaroopch.com/notes/python/) is a beginner's - tutorial for the Python language. - * Google put together a great compilation of materials and subjects you should read and learn from if you want to be a [professional programmer](https://www.google.com/about/careers/students/guide-to-technical-development.html). diff --git a/content/pages/01-introduction/09-best-python-videos.markdown b/content/pages/01-introduction/09-best-python-videos.markdown index 85000438e..14be9bccd 100644 --- a/content/pages/01-introduction/09-best-python-videos.markdown +++ b/content/pages/01-introduction/09-best-python-videos.markdown @@ -163,7 +163,7 @@ like [PyCon US](https://us.pycon.org/) and Python language. * PyCon US videos from - [2020](https://www.youtube.com/c/PyCon2020/videos), + [2020](https://www.youtube.com/playlist?list=PL2Uw4_HvXqvbpFquYIE57BEAqkQWk-iFg), [2019](https://www.youtube.com/channel/UCxs2IIVXaEHHA4BtTiWZ2mQ/videos), [2018](https://www.youtube.com/channel/UCsX05-2sVSH7Nx3zuk3NYuQ/videos), [2017](https://www.youtube.com/channel/UCrJhliKNQ8g0qoE_zvL8eVg/videos), diff --git a/content/pages/01-introduction/10-best-python-podcasts.markdown b/content/pages/01-introduction/10-best-python-podcasts.markdown index f0181a269..a2c528fee 100644 --- a/content/pages/01-introduction/10-best-python-podcasts.markdown +++ b/content/pages/01-introduction/10-best-python-podcasts.markdown @@ -42,16 +42,16 @@ listen and learn. [web application development](/web-development.html) in Python using the [Django web framework](/django.html). +* [The Real Python Podcast](https://realpython.com/podcasts/rpp/) is a weekly + podcast with interviews, coding tips, and conversation with guests from the + Python community. + * [Teaching Python](https://www.teachingpython.fm/) is a podcast by two teachers about their adventures teaching middle school computer science, problem solving, handling failure, frustration, and success with teaching Python programming. -* Professor Philip Guo has a video podcast called - [PG Podcast](https://podcasts.apple.com/us/podcast/philip-guo-podcasts-pgbovine-net/id1276072242), - which typically covers Python subjects. -* [The Real Python Podcast](https://realpython.com/podcasts/rpp/) is a weekly podcast with interviews, coding tips, and conversation with guests from the Python community. ## Favorite podcast episodes Here are a list of my favorite episodes from various Python podcasts before diff --git a/content/pages/02-development-environments/00-development-environments.markdown b/content/pages/02-development-environments/00-development-environments.markdown index cb27c7ee4..4b7dc8b94 100644 --- a/content/pages/02-development-environments/00-development-environments.markdown +++ b/content/pages/02-development-environments/00-development-environments.markdown @@ -121,12 +121,6 @@ configuration as a starting point and customize it from there. [which version of Python to use](/python-2-or-3.html) and adding [code metrics](/code-metrics.html) libraries for checking syntax. -* [Three Ways to Install Python on your Windows Computer](http://blog.yhat.com/posts/installing-python-on-windows.html) - provides multiple avenues for Windows users to get Python on their machine - before setting up the rest of their development environment. Unlike - macOS and Linux, the Windows [operating system](/operating-systems.html) - does not include Python with its default installation. - * [PyCharm: The Good Parts](http://nafiulis.me/pycharm-the-good-parts-i.html) shows you how to be more efficient and productive with that IDE if it's your choice for writing Python code. diff --git a/content/pages/02-development-environments/02-vim.markdown b/content/pages/02-development-environments/02-vim.markdown index bb3895f79..f37c04a5d 100644 --- a/content/pages/02-development-environments/02-vim.markdown +++ b/content/pages/02-development-environments/02-vim.markdown @@ -108,10 +108,6 @@ you are unfamiliar with. * [Vim and Python](https://justin.abrah.ms/vim/vim_and_python.html) shows and explains many Python-specific .vimrc options. -* [Vim as a Python IDE](http://liuchengxu.org/posts/use-vim-as-a-python-ide/) - shows a slew of plugins and configuration options for coding with Python - in Vim. - * This [repository's folder with Vimrc files](https://github.com/amix/vimrc/tree/master/vimrcs) has example configurations that are well commented and easy to learn from. @@ -171,11 +167,6 @@ to get started with these tutorials. * [Vim Adventures](http://vim-adventures.com/) is a cute, fun browser-based game that helps you learn Vim commands by playing through the adventure. -* In [Vim: revisited](http://mislav.uniqpath.com/2011/12/vim-revisited/) the - author explains his on-again off-again relationship with using Vim. He then - shows how he configures and uses the editor so it sticks as his primary - code editing tool. - * [Things About Vim I Wish I Knew Earlier](https://blog.petrzemek.net/2016/04/06/things-about-vim-i-wish-i-knew-earlier/) explores the lessons one developer learned while exclusively using Vim for several years. The author includes using relative instead of absolute diff --git a/content/pages/02-development-environments/04-sublime-text.markdown b/content/pages/02-development-environments/04-sublime-text.markdown index 841687737..196df3b51 100644 --- a/content/pages/02-development-environments/04-sublime-text.markdown +++ b/content/pages/02-development-environments/04-sublime-text.markdown @@ -68,10 +68,6 @@ links should get your editor customized with linters, is a spectacular tutorial that covers installing Sublime Text and configuring a multitude of helpful Python programming plugins. -* [Sublime Text 3 Heaven](https://www.kennethreitz.org/essays/sublime-text-3-heaven) - is a quick overview of the extensions, packages and bonus toys that - one developer uses for his own Sublime Text development setup. - * [Sublime Tutor](https://sublimetutor.com/) is an interactive in-editor keyboard shortcuts tutorial that plugs into Sublime so you can learn and become more productive as you use the editor. diff --git a/content/pages/02-development-environments/05-pycharm.markdown b/content/pages/02-development-environments/05-pycharm.markdown index c6810f26c..c464d6e87 100644 --- a/content/pages/02-development-environments/05-pycharm.markdown +++ b/content/pages/02-development-environments/05-pycharm.markdown @@ -29,10 +29,6 @@ Python code. is a solid discussion thread with different developers' perspectives on using PyCharm for coding their applications. -* [How to Get Started with PyCharm and Have a Productive Python IDE](https://pedrokroger.net/getting-started-pycharm-python-ide/) - covers the basics of configuring PyCharm for running code within - virtualenvs, macros, using the console and code completion. - * [Using PyCharm with Pyramid](https://docs.pylonsproject.org/projects/pyramid-cookbook/en/latest/development_tools/pycharm.html) is specific to developing and debugging [with the Pyramid web framework](/pyramid.html). diff --git a/content/pages/02-development-environments/08-bash-shell.markdown b/content/pages/02-development-environments/08-bash-shell.markdown index 7fd89c38e..22c7877bc 100644 --- a/content/pages/02-development-environments/08-bash-shell.markdown +++ b/content/pages/02-development-environments/08-bash-shell.markdown @@ -219,7 +219,7 @@ to try to avoid as you work with the shell or write scripts. their destructive abilities by reading through the descriptions provided by the author. -* [Faster bash startup](https://danpker.com/posts/2020/faster-bash-startup/) +* [Faster bash startup](https://danpker.com/posts/faster-bash-startup/) and [Even faster bash startup](https://work.lisk.in/2020/11/20/even-faster-bash-startup.html) are two great tutorials that will save you a bunch of time if you frequently diff --git a/content/pages/02-development-environments/12-tmux.markdown b/content/pages/02-development-environments/12-tmux.markdown index 1813d8f4f..4b95fba6c 100644 --- a/content/pages/02-development-environments/12-tmux.markdown +++ b/content/pages/02-development-environments/12-tmux.markdown @@ -38,8 +38,6 @@ easier to use many shells at once and attaching to both local and remote * [Using tmux Properly](http://danielallendeutsch.com/blog/16-using-tmux-properly.html) -* [Differences between tmux vs screen](https://wtanaka.com/node/8136) - * [The Power Of tmux Hooks](https://devel.tech/tips/n/tMuXz2lj/the-power-of-tmux-hooks/) * There are a slew of "cheat sheets" for tmux out there, here are a few diff --git a/content/pages/02-development-environments/15-application-dependencies.markdown b/content/pages/02-development-environments/15-application-dependencies.markdown index a01f2eba9..927a3720d 100644 --- a/content/pages/02-development-environments/15-application-dependencies.markdown +++ b/content/pages/02-development-environments/15-application-dependencies.markdown @@ -39,9 +39,6 @@ all yourself. A few of the best collections of Python libraries are shows the open source Python projects trending today, this week, and this month. -* This list of [20 Python libraries you can’t live without](http://freepythontips.wordpress.com/2013/07/30/20-python-libraries-you-cant-live-without/) - is a wide-ranging collection from data analysis to testing tools. - * Wikipedia actually has an extensive [page dedicated to Python libraries](http://en.wikipedia.org/wiki/List_of_Python_software) grouped by categories. @@ -161,16 +158,6 @@ so far to get up to speed on building and releasing your own packages. provides a collection of resources to understand how to package and distribute Python code libraries. -* [Alice in Python projectland](https://veekaybee.github.io/2017/09/26/python-packaging/) - is an amazing post that takes the reader from simple Python script - into a complete Python package. - -* [Perils of packaging](https://malramsay.com/post/perils_of_packaging/) - covers several edge cases that come up when trying to put together - pieces like Travis CI, PyPI and conda. The post walks through the errors - and how to get around them until they are smoothed out by updates to the - tools. - * [How to Publish Your Package on PyPI](https://blog.jetbrains.com/pycharm/2017/05/how-to-publish-your-package-on-pypi/) is for developers who have created a code library they would like to share and make installable for other developers. @@ -230,10 +217,6 @@ developers. dependency management tools including newer ones such as pipenv and Poetry. -* Occasionally arguments about using Python's dependency manager versus - one of Linux's dependency managers comes up. This provides - [one perspective on that debate](http://notes.pault.ag/debian-python/). - * [Open source trust scaling](http://lucumr.pocoo.org/2016/3/24/open-source-trust-scaling/) is a good piece for the [Python community](/python-community.html) (and other programming communities) that is based on the diff --git a/content/pages/02-development-environments/18-source-control.markdown b/content/pages/02-development-environments/18-source-control.markdown index 83913183d..c96745a51 100644 --- a/content/pages/02-development-environments/18-source-control.markdown +++ b/content/pages/02-development-environments/18-source-control.markdown @@ -194,9 +194,9 @@ implementation. free online version of the O'Reilly [Version Control with Subversion](https://www.amazon.com/dp/B002SR2QIW/) book. -* [How to use Subversion (SVN)](https://deveo.com/svn-tutorial/) lays out - the basic concepts and provides the first few steps for getting started - tracking files. +* [How to Host SVN Repositories](https://www.perforce.com/blog/vcs/how-host-subversion-svn) + lays out the basic concepts and provides the first few steps for getting + started tracking files. * [10 Most Used SVN Commands with Examples](http://www.thegeekstuff.com/2011/04/svn-command-examples/) is a good refresher list if you've used SVN in the past but it has been diff --git a/content/pages/02-development-environments/19-git.markdown b/content/pages/02-development-environments/19-git.markdown index ae0fab9b8..5bffca262 100644 --- a/content/pages/02-development-environments/19-git.markdown +++ b/content/pages/02-development-environments/19-git.markdown @@ -174,10 +174,6 @@ workflow. These resources will come in handy for specific Git subjects. interactive mode or on its own with `git bisect run` to find the problematic code commit that needs to be fixed. -* [How Microsoft uses Git](https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/use-git-microsoft) - gives a high-level overview of their repository structure and hosting - at the extremely large scale organization. - * [GitTips](https://git.wiki.kernel.org/index.php/GitTips) is a list of pro tips to clean up common issues and how to dive through Git history to find specific text. @@ -244,10 +240,6 @@ minimize merge conflicts. why at GitHub they do not use the git-flow model and provides an alternative that solves some of the issues they found with git-flow. -* [Git Workflows That Work](http://blog.endpoint.com/2014/05/git-workflows-that-work.html) - is a helpful post with diagrams to show how teams can create a Git workflow - that will help their development process. - * [Comparing workflows](https://www.atlassian.com/git/tutorials/comparing-workflows) provides a slew of examples for how developers on a team can handle merge conflicts and other situations that commonly arise when using Git. diff --git a/content/pages/03-data/00-data.markdown b/content/pages/03-data/00-data.markdown index a2c996e10..39a767500 100644 --- a/content/pages/03-data/00-data.markdown +++ b/content/pages/03-data/00-data.markdown @@ -12,7 +12,7 @@ subsections, including (in no particular order): * data processing / wrangling * machine learning -* data analysis +* [data analysis](/data-analysis.html) * [visualization](/data-visualization.html) * geospatial mapping * persistence via [relational databases](/databases.html) and @@ -154,6 +154,17 @@ sets. have evolved over the past 20ish years based on his first-hand experience as a leader and member in that community. +* The State of [Python Speech Recognition](https://www.assemblyai.com/blog/the-state-of-python-speech-recognition-in-2021) + in 2021 is a practical overview of a specific area in data: extracting text + from voice recording data. Looking at verticals like this one can make it + easier to understand changes that are occurring in some parts of data and + programming that could be applied to other areas. + +* [Automated Data Wrangling](https://catalyst.coop/2021/05/23/automated-data-wrangling/) + covers cleaning, labeling, and automating the bunch of activities + that are typically necessary before analysis and data usage can + begin for a project. + * The [Open Source Data Science Masters](http://datasciencemasters.org/) is a well-crafted free curriculum and set of resources for students who want to learn both the theory and technologies for working with data. diff --git a/content/pages/03-data/01-databases.markdown b/content/pages/03-data/01-databases.markdown index 5eed45185..bad071a2c 100644 --- a/content/pages/03-data/01-databases.markdown +++ b/content/pages/03-data/01-databases.markdown @@ -100,7 +100,7 @@ Find out about Python applications with a MySQL backed on the dedicated To work with a relational database using Python, you need to use a code library. The most common libraries for relational databases are: -* [psycopg2](http://initd.org/psycopg/) +* [psycopg](https://www.psycopg.org/) ([source code](https://github.com/psycopg/psycopg2)) for PostgreSQL. @@ -112,8 +112,6 @@ library. The most common libraries for relational databases are: * [cx\_Oracle](https://oracle.github.io/python-cx_Oracle/index.html) for Oracle Database ([source code](https://github.com/oracle/python-cx_Oracle)). - Oracle moved their - [open source driver code from SourceForge to GitHub in 2017](https://blogs.oracle.com/developers/oracle-database-python-driver-now-on-github). SQLite support is built into Python 2.7+ and therefore a separate library @@ -183,7 +181,7 @@ speed on SQL if you have never previously used it. elaborates on one of the trickiest parts of writing SQL statements that bridge one or more tables: the `JOIN`. -* [Writing better SQL](http://www.craigkerstiens.com/2016/01/08/writing-better-sql/) +* [Writing more legible SQL](https://www.craigkerstiens.com/2016/01/08/writing-more-legible-sql/) is a short code styling guide to make your queries easier to read. * [SQL Intermediate](https://www.dataquest.io/blog/sql-intermediate/) is a @@ -265,7 +263,7 @@ speed on SQL if you have never previously used it. 1. Install PostgreSQL on your server. Assuming you went with Ubuntu run ``sudo apt-get install postgresql``. -1. Make sure the [psycopg2](http://initd.org/psycopg/) library is in your +1. Make sure the [psycopg](https://www.psycopg.org/) library is in your application's dependencies. 1. Configure your web application to connect to the PostgreSQL instance. diff --git a/content/pages/03-data/02-postgresql.markdown b/content/pages/03-data/02-postgresql.markdown index 26954a7f8..5e700895a 100644 --- a/content/pages/03-data/02-postgresql.markdown +++ b/content/pages/03-data/02-postgresql.markdown @@ -57,7 +57,7 @@ architecture. To work with relational databases in Python you need to use a database driver, which is also referred to as a database connector. The most common driver library for working with PostgreSQL is -[psycopg2](http://initd.org/psycopg/). There is +[psycopg](https://www.psycopg.org/). There is [a list of all drivers on the PostgreSQL wiki](https://wiki.postgresql.org/wiki/Python), including several libraries that are no longer maintained. If you're working with the @@ -126,13 +126,7 @@ walkthroughs I've read. * This article explains how and why PostgreSQL can handle [full text searching](http://blog.lostpropertyhq.com/postgres-full-text-search-is-good-enough/) - for many use cases. If you're going down this route, read - [this blog post that explains how one developer implemented PostgreSQL full text search with SQLAlchemy](http://blog.garage-coding.com/2015/12/18/postgres-fulltext-search.html). - -* [django-postgres-copy](http://django-postgres-copy.californiacivicdata.org/en/latest/) - is a tool for bulk loading data into a PostgreSQL database based on Django models. - [Say hello to our new open-source software for loading bulk data into PostgreSQL](http://www.californiacivicdata.org/2015/07/17/hello-django-postgres-copy/) - is an introduction to using the tool in your own projects. + for many use cases. * [How to speed up tests in Django and PostgreSQL](http://nemesisdesign.net/blog/coding/how-to-speed-up-tests-django-postgresql/) explains some hacks for making your schema migration-backed run quicker. diff --git a/content/pages/03-data/04-sqlite.markdown b/content/pages/03-data/04-sqlite.markdown index d5aee3aaa..5c7f3fd00 100644 --- a/content/pages/03-data/04-sqlite.markdown +++ b/content/pages/03-data/04-sqlite.markdown @@ -155,11 +155,6 @@ you are having with SQLite rather than going through a general tutorial. [object-relational mapper (ORM)](/object-relational-mappers-orms.html) to implement virtual tables and aggregates on top of SQLite. -* [Use SQLite with Django On AWS Lambda with Zappa](https://blog.zappa.io/posts/use-sqlite-with-django-on-aws-lambda-with-zappa) - provides an example `dev_settings.py` file for - locally testing a [Django](/django.html) application intended for - [AWS Lambda](/aws-lambda.html). - * [SQLite Database Authorization and Access Control with Python](http://charlesleifer.com/blog/sqlite-database-authorization-and-access-control-with-python/) covers how to control access to the SQLite database connection and file even though SQLite normally allows unauthorized access by design. diff --git a/content/pages/03-data/05-object-relational-mappers.markdown b/content/pages/03-data/05-object-relational-mappers.markdown index df5ce8bb0..6451908e8 100644 --- a/content/pages/03-data/05-object-relational-mappers.markdown +++ b/content/pages/03-data/05-object-relational-mappers.markdown @@ -64,7 +64,7 @@ there was a pressing reason. Python ORM libraries are not required for accessing relational databases. In fact, the low-level access is typically provided by another library called a *database connector*, such as -[psycopg](http://initd.org/psycopg/) (for PostgreSQL) +[psycopg](https://www.psycopg.org/) (for PostgreSQL) or [MySQL-python](https://pypi.org/project/MySQL-python/1.2.5) (for MySQL). Take a look at the table below which shows how ORMs can work with different web frameworks and connectors and relational databases. diff --git a/content/pages/03-data/11-redis.markdown b/content/pages/03-data/11-redis.markdown index c05d04741..3600ddae2 100644 --- a/content/pages/03-data/11-redis.markdown +++ b/content/pages/03-data/11-redis.markdown @@ -84,10 +84,6 @@ Redis should be customized out of its default configuration to secure it against unauthorized and unauthenticated users. These resources provide some advice on Reids security and guarding against data breaches. -* [Pentesting Redis servers](http://averagesecurityguy.info/2015/09/17/pentesting-redis-servers/) - shows that security is important not only on your application but also - the databases you're using as well. - * Redis, just as with any relational or NoSQL database, needs to be secured based on [security guidelines](http://www.antirez.com/news/96). There is also a post where the main author of Redis diff --git a/content/pages/03-data/12-mongodb.markdown b/content/pages/03-data/12-mongodb.markdown index 1e9f15161..665144fec 100644 --- a/content/pages/03-data/12-mongodb.markdown +++ b/content/pages/03-data/12-mongodb.markdown @@ -99,11 +99,6 @@ security controls so make sure to lock down your instances. installing and using MongoDB on your own instance. The post covers authentication, SSL and firewalls. -* [Securing MongoDB using Let's Encrypt certificate](https://zohaib.me/securing-mongodb-using-lets-encrypt/) - gives a configuration that encrypts that traffic coming from and - going to your MongoDB instances using free - [Let's Encrypt certificates](https://letsencrypt.org/). - * This 4 post securing MongoDB series covers [Data Security Requirements for Regulatory Compliance](https://www.mongodb.com/blog/post/securing-mongodb-part-1-data-security-requirements-for-regulatory-compliance), [Database Access Control](https://www.mongodb.com/blog/post/securing-mongodb-part-2-database-access-control), @@ -111,11 +106,6 @@ security controls so make sure to lock down your instances. and [Environmental Control & Database Management](https://www.mongodb.com/blog/post/securing-mongodb-part-4-environmental-control-and-database-management). -* Lightweight Directory Access Protocol (LDAP) is common in many - established company environments for security. This post on - [How to Configure LDAP Authentication for MongoDB](https://www.mongodb.com/blog/post/how-to-configure-LDAP-authentication-for-mongodb) - goes over how to authenticate users via LDAP who are using MongoDB. - ### Python with MongoDB resources MongoDB is straightforward to use in a Python application when a driver diff --git a/content/pages/03-data/16-pandas.markdown b/content/pages/03-data/16-pandas.markdown index 9ac2c3bbe..a0750eaae 100644 --- a/content/pages/03-data/16-pandas.markdown +++ b/content/pages/03-data/16-pandas.markdown @@ -82,10 +82,6 @@ is a data structures and analysis library. [Tabula](https://tabula.technology/) library that makes it easier to extract table data from PDF files. -* [Analyzing Browser History Using Python and Pandas](https://applecrazy.github.io/blog/2017-11-12/analyzing-browser-hist-using-python) - shows how to take data from Google Chrome and start to visualize it - with pandas and [matplotlib](/matplotlib.html). - * [Time Series Forecast Case Study with Python: Monthly Armed Robberies in Boston](https://machinelearningmastery.com/time-series-forecast-case-study-python-monthly-armed-robberies-boston/) walks through the data wrangling, analysis and visualization steps with a public data set of murders in Boston from 1966 to 1975. This diff --git a/content/pages/03-data/17-scipy-numpy.markdown b/content/pages/03-data/17-scipy-numpy.markdown index 62ec16355..00763c2d0 100644 --- a/content/pages/03-data/17-scipy-numpy.markdown +++ b/content/pages/03-data/17-scipy-numpy.markdown @@ -44,11 +44,6 @@ following resources are broader walkthroughs for the SciPy ecosystem: plotter. This is a very cool example project that ties together the scientific world and the art world. -* [Lectures in Quantitative Economics: SciPy](https://lectures.quantecon.org/py/scipy.html) - provides a good overview of SciPy compared to the specific NumPy - project, as well as explanations for the wrappers SciPy provides - over lower-level FORTRAN libraries. - * [A plea for stability in the SciPy ecosystem](http://blog.khinsen.net/posts/2017/11/16/a-plea-for-stability-in-the-scipy-ecosystem/) presents concerns from one scientist's perspective about how fast the Python programming ecosystem changes and that code can become backwards diff --git a/content/pages/03-data/18-data-visualization.markdown b/content/pages/03-data/18-data-visualization.markdown index af18aada8..e56c2b2dc 100644 --- a/content/pages/03-data/18-data-visualization.markdown +++ b/content/pages/03-data/18-data-visualization.markdown @@ -91,11 +91,6 @@ Sometimes you need inspiration from other sources to figure out what you want to build. The following links have made me excited about data visualization and gave me ideas for what to build. -* [Roads to Rome](http://roadstorome.moovellab.com/) is a beautiful - visualization showing the data behind the expression "all roads lead - to Rome" and whether or not there is a "Rome" central city in every - country. - * [Monarchs](https://thebackend.dev/monarchs/) is a wonderful 1,000 year history visual of European rulers. The developer also wrote an in-depth article on @@ -107,10 +102,6 @@ visualization and gave me ideas for what to build. and dark sides, the main characters, various bits about the Force and other data extracted from the movies. -* [Big League Graphs](https://bigleaguegraphs.com/) presents a bunch of - creative ways to view data for sports such as basketball, baseball and - hockey. - * [What do numbers look like?](https://johnhw.github.io/umap_primes/index.md.html) is a Python 3 dimensional visualization of millions of integers, colored by special factors such as prime and Fibonacci numbers. diff --git a/content/pages/03-data/19-bokeh.markdown b/content/pages/03-data/19-bokeh.markdown index 668538d90..22f8d55cc 100644 --- a/content/pages/03-data/19-bokeh.markdown +++ b/content/pages/03-data/19-bokeh.markdown @@ -61,12 +61,6 @@ basic syntax will change as the library's API is not yet stable. an appropriate format then explains the code that uses Bokeh to visualize it. -* [Data is beautiful: Visualizing Roman imperial dynasties](http://machineloveus.com/data-is-beautiful-visualizing-roman-imperial-dynasties/) - provides a walkthrough for creating a gorgeous visualization based on - historical Roman data. The post is about more than just the visual, it also - goes into the ideation, data wrangling and analysis phases that came - before using Bokeh to show the results. - * [Visualizing with Bokeh](https://programminghistorian.org/en/lessons/visualizing-with-bokeh) gives a detailed explanation with the code for number Bokeh visuals you can output while working with a [pandas](/pandas.html) data set. @@ -105,13 +99,6 @@ basic syntax will change as the library's API is not yet stable. project has the code to create a simple chart with Bokeh and [Flask](/flask.html). -* [Bokeh vs Dash — Which is the Best Dashboard Framework for Python?](https://blog.sicara.com/bokeh-dash-best-dashboard-framework-python-shiny-alternative-c5b576375f7f) - contains a single project that was written in both Dash and Bokeh. The - author gives his subjective view on the implementation difficulty - although the web application only contained a single type of data - visualization so it is hard to drawn any real conclusions from his - opinion. - * [Realtime Flight Tracking with Pandas and Bokeh](https://www.geodose.com/2019/01/realtime-flight-tracking-pandas-bokeh-python.html) provides a great example of combining [pandas](/pandas.html) for structuring data with Bokeh for visualization. diff --git a/content/pages/03-data/21-matplotlib.markdown b/content/pages/03-data/21-matplotlib.markdown index 14f4d2f64..9c60d3740 100644 --- a/content/pages/03-data/21-matplotlib.markdown +++ b/content/pages/03-data/21-matplotlib.markdown @@ -21,10 +21,6 @@ toolkits. is an awesome getting started tutorial that breaks through the confusing beginner steps so you can quick start using the plotting library. -* [Web Scraping With Python: Scrapy, SQL, Matplotlib To Gain Web Data Insights](http://www.scrapingauthority.com/python-scrapy-mysql-and-matplotlib-to-gain-web-data-insights/) - is a long and comprehensive tutorial that walks through obtaining, - cleaning and visualizing data. - * [Matplotlib Cheat Sheet: Plotting in Python](https://www.datacamp.com/community/blog/python-matplotlib-cheat-sheet) contains some handy snippets of code to perform common plotting operations in Matplotlib. diff --git a/content/pages/03-data/25-oracle.markdown b/content/pages/03-data/25-oracle.markdown new file mode 100644 index 000000000..bf6b3cdc0 --- /dev/null +++ b/content/pages/03-data/25-oracle.markdown @@ -0,0 +1,165 @@ +title: Oracle +category: page +slug: Oracle +sortorder: 0325 +toc: False +sidebartitle: Oracle +meta: Oracle Database is an enterprise relational database management system. + + +[Oracle Database](http://www.oracle.com/) is an enterprise +[relational database](/databases.html). It can run transaction processing, +data warehousing, and multi-model database workloads such as machine +learning, spatial, and graph analysis. Recent versions of Oracle Database +also added support for JSON and blockchain use cases, and the software +can be run in on-premise, cloud or hybrid environments. + +Oracle logo. + + +## How does Oracle fit with Python? +The Python community and Oracle have a long history. The excellent Python Database API-compliant "cx_Oracle" interface for Oracle Database was first created by the user community in 1998 and is now being enhanced and maintained by Oracle. The [cx_Oracle](https://oracle.github.io/python-cx_Oracle/) module also underpins the [Oracle Machine Learning for Python](https://www.youtube.com/watch?v=P861m__PEMQ) engine. Oracle's high-performance GraalVM framework supports an implementation of Python called [GraalPython](https://github.com/oracle/graalpython). + + +## Why is Oracle Database a great choice? +Oracle Database is cross-platform, supporting multiple hardware platforms and various operating systems. Developers and companies of all sizes rely on its proven industry-leading performance, scalability, reliability, and security. +As data volumes rise exponentially, new data types and data models are required to support modern applications. Oracle Database supports the following data types at no extra cost: + +* [JSON](https://docs.oracle.com/en/database/oracle/oracle-database/19/adjsn/index.html) +* [Blockchain](https://docs.oracle.com/en/database/oracle/oracle-database/21/nfcon/details-oracle-blockchain-table-282449857.html) +* [XML](https://www.oracle.com/database/technologies/appdev/xmldb.html) +* [Object](https://docs.oracle.com/database/121/ADOBJ/adobjint.htm#ADOBJ00101) +* [Graph](https://www.oracle.com/database/graph/) +* [Spatial](https://www.oracle.com/database/spatial/) +* [Time Series](https://docs.oracle.com/en/database/oracle/oracle-database/19/dmcon/time-series.html) +* Relational + +With support for scale-out database clusters, sharded distributed systems, and disaster recovery with continuous application availability, there is no shortage of features to guarantee the Database continues to run uninterrupted 24/7. + +Oracle makes its enterprise-class database readily available to developers with its free on-premises edition Oracle Database XE or on the Oracle public cloud with an Always Free Cloud account. In addition, Oracle Autonomous Database is a popular choice for developers as no database management or tuning is required, leaving developers to do what they do best – writing code for their applications. + + +## Connecting to Oracle Database with Python +As with any database, applications require a connector or driver to connect to the Oracle Database. The Python DB API-compliant [cx_Oracle](https://github.com/oracle/python-cx_Oracle) interface provides developers access to standard and advanced Oracle Database features, such as SQL execution and document storage APIs. It also gives users access to network traffic encryption capabilities and Oracle's leading high availability features. + +[Code examples](https://oracle.github.io/python-cx_Oracle/samples/tutorial/Python-and-Oracle-Database-Scripting-for-the-Future.html) and free workshops such as the introductory [Python and Oracle for Developers Workshop](https://apexapps.oracle.com/pls/apex/dbpm/r/livelabs/view-workshop?wid=766) and a full-stack development workshop using [Python with SQLAlchemy to Oracle Database](https://apexapps.oracle.com/pls/apex/dbpm/r/livelabs/view-workshop?wid=911&clear=180&session=16650643444916) are available. + +cx Oracle driver. + +You can use many Python frameworks and [object-relational mappers (ORMs)](/object-relational-mappers-orms.html) with Oracle Database. ORMs abstract the tables and objects in a relational database to objects that Python developers can manipulate and operate on. [SQLAlchemy](/sqlalchemy.html) and Django are popular ORMs. SQLAlchemy is used by Pandas, which is very popular with Oracle users. +The table below shows the relationship between web framework, ORM, driver, and the Oracle Database. + +Examples of how varying Python ORMs can work with Oracle and the cx Oracle connector. + +Learn more about +[Python ORMs on that dedicated topic page](/object-relational-mappers-orms.html). + +ORMs provide a familiar programming model for Python developers, but sometimes you want that extra performance and operate closer to SQL objects. Oracle cx_Oracle offers several [functions](https://oracle.github.io/python-cx_Oracle/samples/tutorial/Python-and-Oracle-Database-Scripting-for-the-Future.html#binding) to deliver that performance. These functions include fetching data, binding data, executing PL/SQL, operating on LOBs, JSON documents, message passing with Oracle Advanced Queuing, and more. + + +## Oracle and Data Safety +According to Gartner, Oracle has one of the [highest data safety ratings](https://www.gartner.com/reviews/market/cloud-database-management-systems/vendor/oracle/product/oracle-database) in the industry, with a wide range of features for data protection and high availability. These features include: + +* [Database encryption](https://www.oracle.com/database/technologies/security/advanced-security.html) + +* [Access control to rows](https://www.oracle.com/database/technologies/security/label-security.html) in a table + +* [Database vault](https://www.oracle.com/database/technologies/security/db-vault.html) to restrict privileges and access + +* [Data redaction, subsetting, and masking](https://www.oracle.com/database/technologies/security/data-masking-subsetting.html) + +* All in one data security service in the Oracle Cloud with [Data Safe](https://www.oracle.com/database/technologies/security/data-safe.html) + +* Oracle also provides free tools such as the [Database Assessment Tool (DBSAT)](https://www.oracle.com/database/technologies/security/dbsat.html) to help you identify and remedy potential vulnerabilities. + +Oracle also provides numerous data recovery features, including: + +* Backup capabilities with [RMAN](https://www.oracle.com/database/technologies/high-availability/rman.html) + +* Restore point features with [Database Flashback](https://www.oracle.com/database/technologies/high-availability/flashback.html) + +* [Application continuity](https://www.oracle.com/database/technologies/high-availability/app-continuity.html) in the event of database failover to a standby + +For an overview of Oracle’s security and high availability architecture, see the following white papers: + +* [Maximum Availability Architecture](https://www.oracle.com/a/tech/docs/maa-onpremises-overview.pdf) (MAA) + +* [Maximum Security Architecture](https://blogs.oracle.com/cloudsecurity/post/oracles-maximum-security-architecture-for-database-security) (MSA) + + +## Python Specific Oracle Database resources +Many quick starts, tutorials, and workshops exist specifically for Python developers using Oracle Database. Below are some of the best ones to start with. + + +###Getting Started +If you are looking for a fast way to get started with Python and Oracle Database, check out these two quick start tutorials. These tutorials walk you through installing and setting up the environment you need to connect Python to Oracle Database. + +* [Quick Start: Developing Python Applications for Oracle Database](https://www.oracle.com/database/technologies/appdev/python/quickstartpythononprem.html) + +* [Quick Start: Developing Python Applications for Oracle Autonomous Database](https://www.oracle.com/database/technologies/appdev/python/quickstartpythononprem.html) + +Once you have done one of these, then continue with the popular [Python and Oracle Database Tutorial: Scripting for the Future](https://oracle.github.io/python-cx_Oracle/samples/tutorial/Python-and-Oracle-Database-Scripting-for-the-Future.html) to dive deeper to master the Python cx_Oracle interface and see how to build great Oracle Database applications. + + +###Using Different Frameworks with Oracle +* [How to Run SQL Queries with Pandas](https://www.oracle.com/news/connect/run-sql-data-queries-with-pandas.html) is a good blog using Pandas for quick and easy data manipulation in Python. + +* [Using Oracle with Pandas in OCI Data Science Notebooks](https://docs.oracle.com/en-us/iaas/tools/ads-sdk/latest/user_guide/loading_data/efficient_use_of_oracle_rdbms_with_ads.html) dives deeper into using Pandas with large datasets in data science applications. + +* [Using SQLAlchemy with Oracle Database](https://docs.sqlalchemy.org/en/14/dialects/oracle.html) provides an excellent toolkit for Python developers using SQLAlchemy as their ORM. + +* [Using Django with Python and Oracle Database](https://www.oracle.com/webfolder/technetwork/tutorials/obe/db/oow10/python_django/python_django.htm) is a tutorial from Oracle and shows the Django Framework with Python to an Oracle Database. + +* [Connecting Pony ORM to the Database](https://docs.ponyorm.org/database.html) is a friendly guide on using Pony with databases. + +* [How to use Python Flask with Oracle Database](https://blogs.oracle.com/opal/post/how-to-use-python-flask-with-oracle-database). + +* [Part 1: Docker for Oracle Database Applications in Node.js and Python](https://blogs.oracle.com/opal/post/part-1-docker-for-oracle-database-applications-in-nodejs-and-python). + +* [Part 2: Docker for Oracle Database Applications in Node.js and Python](https://blogs.oracle.com/opal/post/part-2-docker-for-oracle-database-applications-in-nodejs-and-python). + +* [Faster JSON with Python cx_Oracle and Oracle Database 21’s new OSON storage format](https://blogs.oracle.com/opal/post/faster-json-with-python-cx_oracle-81-and-oracle-database-21s-new-oson-storage-format). + + +###Workshops +The following hands-on, free workshops provide step-by-step instructions and walkthroughs in a live environment. + +* [Use Python with Oracle Database 19c](https://apexapps.oracle.com/pls/apex/dbpm/r/livelabs/view-workshop?wid=635&clear=180&session=3484600041895) is an Oracle LiveLabs workshop that shows how to write Python code to connect to and read data from an Oracle Database, including JSON data. + +* [Python and Oracle for Developers](https://apexapps.oracle.com/pls/apex/dbpm/r/livelabs/workshop-attendee-2?p210_workshop_id=766&p210_type=2&session=3484600041895) is an Oracle LiveLabs workshop that explores the features of the Python cx_Oracle interface for Oracle Database, including efficient techniques for connection management and statement handling. + +* [Full Stack Development using Python and deployment via OKE](https://apexapps.oracle.com/pls/apex/dbpm/r/livelabs/view-workshop?wid=911&clear=180&session=3484600041895) is an Oracle LiveLabs workshop that explores how to build and deploy a simple cloud-native application using the most common frameworks and the Oracle Cloud Infrastructure services. + + +## Cloud Development with Oracle Database +The following resources are good starting points for those looking to build applications in the Oracle Cloud and deploy applications in Docker containers and Kubernetes. + +* [The Complete Guide To Getting Up And Running With Docker And Kubernetes On The Oracle Cloud](https://blogs.oracle.com/developers/post/the-complete-guide-to-getting-up-and-running-with-docker-and-kubernetes-on-the-oracle-cloud). + +* [Oracle Cloud Blog](https://www.oc-blog.com/) has lots of interesting information on different aspects of Oracle Cloud. + +For developers looking to focus on application development in the Oracle Cloud and not have to worry about managing the Oracle Database, the Autonomous Database is a good choice. All management, including patching and upgrades, scalability, and security, are entirely autonomous. The following resources offer you a glimpse of its capabilities. + +* [Julien Dontcheff’s Database Blog](https://juliandontcheff.wordpress.com/category/autonomous/) is a good collection of technical posts with the Autonomous Database. + +* [SQL Maria](https://sqlmaria.com/category/autonomous-database/) also has some excellent posts on all things Oracle Database including Autonomous. + +* [An Introduction to Autonomous Database](https://questoraclecommunity.org/learn/blogs/oracles-autonomous-database-an-introduction/) gives you a good overview. + +* [Autonomous Database for researchers](https://blogs.oracle.com/research/post/a-roadmap-of-oracle-autonomous-database-benefits-for-research) is a good blog with details on some autonomous features. + + +##General Oracle Database Resources +Here are some Oracle tutorials and resources not specific to Python that can help you take advantage of the Oracle Database features. + +* [Oracle Technical Architecture](https://www.oracle.com/webfolder/technetwork/tutorials/architecture-diagrams/18/technical-architecture/database-technical-architecture.html) is from Oracle and has nice visuals and short paragraphs on the architecture of the Oracle Database. + +* [Oracle Database Internals](https://databaseinternalmechanism.com/oracle-database-internals/) is an excellent post explaining the architecture of the Oracle Database. + +* This [Oracle Performance Tuning](https://blog.quest.com/oracle-performance-tuning-a-5-step-approach-to-optimized-performance/) blog has a 5-step approach to tuning Oracle. + +* [Oracle RAC](https://databaseinternalmechanism.com/oracle-rac/) is another good post on the concepts of RAC, Oracle’s Real Application Cluster software for database high availability. + +* The [Oracle Database Security](https://www.oracle.com/database/technologies/security.html) web page has lots of information on Oracle’s solutions for security called “defense in depth.” + +* This is a good post on the [Top 5 Reasons to choose Oracle](https://www.dbta.com/Editorial/News-Flashes/Top-5-Reasons-to-Use-an-Oracle-Database-144191.aspx) for a production database. diff --git a/content/pages/04-web-development/00-web-development.markdown b/content/pages/04-web-development/00-web-development.markdown index 29b3fb98b..919cc6c94 100644 --- a/content/pages/04-web-development/00-web-development.markdown +++ b/content/pages/04-web-development/00-web-development.markdown @@ -71,12 +71,6 @@ world. [data bases](/databases.html), [task queues](/task-queues.html), [caching](/caching.html) and several other critical concepts. -* The [Evolution of the Web](http://www.evolutionoftheweb.com/) visualizes - how web browsers and related technologies have changed over time as well as - the overall growth of the Internet in the amount of data transferred. Note - that the visualization unfortunately stops around the beginning of 2013 but - it's a good way to explore what happened in the first 24 years. - * [What happens when?](https://github.com/alex/what-happens-when) is an incredibly detailed answer to the questions "What happens when you type google.com into your browser's address box and press enter?" that @@ -92,19 +86,6 @@ world. the creation of the URL. This is a great read that provides historical context for why things are the way they are with the web. -* [Web app checklist](http://dhilipsiva.com/webapp-checklist/) presents - good practices that developers building and [deploying](/deployment.html) - web applications should follow. Don't worry about having every single - one of these recommendations implemented before getting your site - live, but it is worthwhile to review the list to make sure there is not - something obvious you can handle in a few minutes that will improve - your site's security, performance or usability. - -* [Web application development is different and better](http://radar.oreilly.com/2014/01/web-application-development-is-different-and-better.html) - provides some context for how web development has evolved from writing - static HTML files into the complex JavaScript client-side applications - produced today. - * [The Browser Hacker's Guide to Instantly Loading Everything](https://www.youtube.com/watch?v=7vUs5yOuv-o) is a spectacular technical talk given by Addy Osmani at JSConf EU 2017 that has great bits of developer knowledge for both beginner and diff --git a/content/pages/04-web-development/02-django.markdown b/content/pages/04-web-development/02-django.markdown index b5a5cea0c..3061d2eee 100644 --- a/content/pages/04-web-development/02-django.markdown +++ b/content/pages/04-web-development/02-django.markdown @@ -121,7 +121,7 @@ These books and tutorials assume that you know the basics of building Django and want to go further to become much more knowledgeable about the framework. -* [2 Scoops of Django](https://www.twoscoopspress.com/collections/django/products/two-scoops-of-django-1-11) +* [2 Scoops of Django](https://www.feldroy.com/books/two-scoops-of-django-3-x) by Daniel Greenfeld and Audrey Roy is well worth the price of admission if you're serious about learning how to correctly develop Django websites. @@ -338,11 +338,6 @@ out how to build your own projects. This is a short list of some real-world example applications, and many more can be found on the [Django example projects and code](/django-code-examples.html) page. -* [Browser calls with Django and Twilio](https://www.twilio.com/docs/howto/walkthrough/browser-calls/python/django) - shows how to build a web app with Django and - [Twilio Client](https://www.twilio.com/client) to turn a user's web - browser into a full-fledged phone. Pretty awesome! - * [Openduty](https://github.com/ustream/openduty) is a website status checking and alert system similar to PagerDuty. diff --git a/content/pages/04-web-development/03-flask.markdown b/content/pages/04-web-development/03-flask.markdown index 2de57acbb..fd7625f0e 100644 --- a/content/pages/04-web-development/03-flask.markdown +++ b/content/pages/04-web-development/03-flask.markdown @@ -295,7 +295,7 @@ about how to working on your codebase. * [Bean Counter](https://github.com/BouncyNudibranch/bean-counter) is an open source Flask app for tracking coffee. -* [FlaskBB](http://flaskbb.org/) is a Flask app for a discussion forum. +* [FlaskBB](https://flaskbb.org/) is a Flask app for a discussion forum. * [psdash](https://github.com/Jahaja/psdash) is an app built with Flask and psutils to display information about the computer it is running on. diff --git a/content/pages/04-web-development/04-bottle.markdown b/content/pages/04-web-development/04-bottle.markdown index 1f40db51c..01e543da4 100644 --- a/content/pages/04-web-development/04-bottle.markdown +++ b/content/pages/04-web-development/04-bottle.markdown @@ -29,7 +29,7 @@ Bottle is awesome for a few web development situations: Prototyping simple ideas is often easier with Bottle than a more opinionated web framework like [Django](/django.html) because Django projects start with a significant amount of boilerplate code. The -[Model-View-Template](https://docs.djangoproject.com/en/1.9/faq/general/#django-appears-to-be-a-mvc-framework-but-you-call-the-controller-the-view-and-the-view-the-template-how-come-you-don-t-use-the-standard-names) +[Model-View-Template](https://docs.djangoproject.com/en/stable/intro/tutorial03/) structure for Django apps within projects makes maintaining projects easier, but it can be cumbersome on starter projects where you're just playing with random ideas so you aren't worried about your diff --git a/content/pages/04-web-development/11-template-engines.markdown b/content/pages/04-web-development/11-template-engines.markdown index f8b7bac06..551a70bfb 100644 --- a/content/pages/04-web-development/11-template-engines.markdown +++ b/content/pages/04-web-development/11-template-engines.markdown @@ -161,6 +161,10 @@ know how they work to aid your debugging. The following resources examine existing template engine design as well as how to build your own engine when that's necessary for your projects. +* [Writing a Jinja-inspired template library in Python](https://notes.eatonphil.com/writing-a-template-library-in-python.html) + walks through how to create your own a simplified version of the + [Jinja](/jinja2.html) template engine as a learning exercise. + * [How a template engine works](https://fengsp.github.io/blog/2016/8/how-a-template-engine-works/) uses the template module in Tornado as an example to step through how a template engine produces output, from parsing the incoming string to diff --git a/content/pages/04-web-development/12-jinja2.markdown b/content/pages/04-web-development/12-jinja2.markdown index 23137b27d..75feb4fdc 100644 --- a/content/pages/04-web-development/12-jinja2.markdown +++ b/content/pages/04-web-development/12-jinja2.markdown @@ -8,17 +8,17 @@ meta: Jinja2 is a template engine written in Python for outputting formats such Jinja, also commonly referred to as -"[Jinja2](http://jinja.pocoo.org/docs/dev/)" to specify the newest +"[Jinja2](https://jinja.palletsprojects.com/en/3.0.x/templates/)" to specify the newest release version, is a Python [template engine](/template-engines.html) used to create HTML, XML or other markup formats that are returned to the user via an HTTP response. -Logo for the Jinja template engine project. +Logo for the Jinja template engine project. ## Why is Jinja2 useful? Jinja2 is useful because it has consistent template tag syntax and the project is cleanly extracted as -[an independent open source project](https://github.com/mitsuhiko/jinja2) so +[an independent open source project](https://github.com/pallets/jinja) so it can be used as a dependency by other code libraries.
Jinja2 is an implementation of the template engines concept. Learn more in the web development chapter or view the table of contents for all topics.
@@ -31,9 +31,8 @@ end a developer can code whatever she wants. ## Jinja2 origin and development The first recorded public released of Jinja2 was in -[2008 with 2.0rc1](http://jinja.pocoo.org/docs/dev/changelog/#version-2-0rc1). -Since then the engine has seen numerous updates and remains in active -development. +2008 with 2.0rc1. Since then the engine has seen numerous updates and +remains under active development. Jinja2 engine certainly wasn't the first template engine. In fact, Jinja2's syntax is inspired by Django's built-in template engine, which was released @@ -72,7 +71,7 @@ open source project author from having to reinvent a new templating style. the template engine. * The official - [Jinja2 template designer documentation](http://jinja.pocoo.org/docs/dev/templates/) + [Jinja2 templates documentation](https://flask.palletsprojects.com/en/2.0.x/tutorial/templates/) is exceptionally useful both as a reference as well as a full read-through to understand how to properly work with template tags. diff --git a/content/pages/04-web-development/17-css.markdown b/content/pages/04-web-development/17-css.markdown index f7d3658b2..d50968d09 100644 --- a/content/pages/04-web-development/17-css.markdown +++ b/content/pages/04-web-development/17-css.markdown @@ -119,6 +119,14 @@ web application's design. provides advice on how to write simpler, easier-to-maintain CSS code to reduce your need to rely on CSS preprocessors and build pipelines. +* [How to Detect Unused CSS or JavaScript](https://javascript.plainenglish.io/detect-unused-css-or-javascript-in-your-code-8d200ef07e50) + explains how to use [Chrome DevTools](https://developer.chrome.com/docs/devtools/) + to analyze a page's CSS and identify parts that are not used. Note that + even though a specific page does not use that CSS (or JS), there might + be another page that uses the same CSS files and *does* use that "unused" + code, so test your pages before and after making the changes to ensure + you did not inadvertently break something else! + * [CSS refresher notes](https://github.com/vasanthk/css-refresher-notes) is incredibly helpful if you've learned CSS in bits and pieces along the way and you now want to fill in the gaps in your knowledge. @@ -149,10 +157,6 @@ web application's design. * [Google's Web Fundamentals class](https://developers.google.com/web/fundamentals/) shows how to create responsive designs and performant websites. -* [Tailoring CSS for performance](http://programming.oreilly.com/2014/04/tailoring-css-for-performance.html) - is an interesting read since many developers do not consider the - implications of CSS complexity in browser rendering time. - * [Can I Use...](http://caniuse.com/) is a compatibility table that shows which versions of browsers implement specific CSS features. @@ -186,16 +190,6 @@ web application's design. talk about tables, because that was the only way to position anything back in the day. -* [30 seconds of CSS](https://30-seconds.github.io/30-seconds-of-css/) - provides short useful code snippets for you to learn from and use for - building your own web applications. - -* [CSS: The bad bits](https://www.joeforshaw.com/blog/css-the-bad-bits-and-how-to-avoid-them) - examines global scope, implicit percentage styling rules and the z-index - which can be difficult to use and require some restraint to ensure they - do not cause issues for the rest of your stylesheet rules as you create - and maintain your frontend. - * [Improving Your CSS with Parker](https://csswizardry.com/2016/06/improving-your-css-with-parker/) shows how to use the static CSS analysis tool [Parker](https://github.com/katiefenn/parker/) to improve your stylesheets. diff --git a/content/pages/04-web-development/27-task-queues.markdown b/content/pages/04-web-development/27-task-queues.markdown index 073759f6e..70ec128c2 100644 --- a/content/pages/04-web-development/27-task-queues.markdown +++ b/content/pages/04-web-development/27-task-queues.markdown @@ -100,13 +100,6 @@ when scaling out a large deployment of distributed task queues. ## Open source examples that use task queues -* Take a look at the code in this open source - [Flask application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/flask) - and - [this Django application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/django) - for examples of how to use and deploy Celery with a Redis broker to - send text messages with these frameworks. - * [flask-celery-example](https://github.com/thrisp/flask-celery-example) is a simple Flask application with Celery as a task queue and Redis as the broker. @@ -128,10 +121,6 @@ when scaling out a large deployment of distributed task queues. is a detailed comparison of Amazon SQS, MongoDB, RabbitMQ, HornetQ and Kafka's designs and performance. -* [Queues.io](http://queues.io/) is a collection of task queue systems with - short summaries for each one. The task queues are not all compatible with - Python but ones that work with it are tagged with the "Python" keyword. - * [Why Task Queues](http://www.slideshare.net/bryanhelmig/task-queues-comorichweb-12962619) is a presentation for what task queues are and why they are needed. @@ -187,10 +176,6 @@ when scaling out a large deployment of distributed task queues. is a straightforward tutorial for setting up the Celery task queue for Django web applications using the Redis broker on the back end. -* [Three quick tips from two years with Celery](https://library.launchkit.io/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb) - provides some solid advice on retry delays, the -Ofair flag and global - task timeouts for Celery. - * [Asynchronous Tasks with Flask and Redis Queue](https://testdriven.io/asynchronous-tasks-with-flask-and-redis-queue) looks at how to configure Redis Queue to handle long-running tasks in a Flask app. diff --git a/content/pages/04-web-development/28-celery.markdown b/content/pages/04-web-development/28-celery.markdown index 5a614855c..6746997d1 100644 --- a/content/pages/04-web-development/28-celery.markdown +++ b/content/pages/04-web-development/28-celery.markdown @@ -7,11 +7,11 @@ sidebartitle: Celery meta: Celery is a task queue for executing work outside a Python web application HTTP request-response cycle. -[Celery](http://www.celeryproject.org/) is a [task queue](/task-queues.html) +[Celery](https://docs.celeryproject.org/) is a [task queue](/task-queues.html) implementation for [Python web applications](/web-development.html) used to asynchronously execute work outside the HTTP request-response cycle. -Celery task queue project logo. +Celery task queue project logo.
Celery is an implementation of the task queue concept. Learn more in the web development chapter or view the table of contents for all topics.
@@ -163,10 +163,6 @@ web framework of your choice. is a detailed walkthrough for setting up Celery with Django (although Celery can also be used without a problem with other frameworks). -* [Introducing Celery for Python+Django](http://www.linuxforu.com/2013/12/introducing-celery-pythondjango/) - provides an introduction to the Celery task queue with Django as the - intended framework for building a web application. - * [Asynchronous Tasks with Falcon and Celery](https://testdriven.io/asynchronous-tasks-with-falcon-and-celery) configures Celery with the [Falcon](/falcon.html) framework, which is less commonly-used in web tutorials. @@ -180,25 +176,14 @@ web framework of your choice. looks at how to configure Celery to handle long-running tasks in a Django app. + ### Celery deployment resources Celery and its broker run separately from your web and WSGI servers so it adds some additional complexity to your [deployments](/deployment.html). The following resources walk you through how to handle deployments and get the right configuration settings in place. -* The "Django in Production" series by - [Rob Golding](https://twitter.com/robgolding63) contains a post - specifically on [Background Tasks](http://www.robgolding.com/blog/2011/11/27/django-in-production-part-2---background-tasks/). - * [How to run celery as a daemon?](https://pythad.github.io/articles/2016-12/how-to-run-celery-as-a-daemon-in-production) is a short post with the minimal code for running the Celery daemon and Celerybeat as system services on Linux. -* [Celery in Production](http://www.caktusgroup.com/blog/2014/09/29/celery-production/) - on the Caktus Group blog contains good practices from their experience - using Celery with RabbitMQ, monitoring tools and other aspects not often - discussed in existing documentation. - -* [Three quick tips from two years with Celery](https://library.launchkit.io/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb) - provides some solid advice on retry delays, the `-Ofair` flag and global - task timeouts for Celery. diff --git a/content/pages/04-web-development/29-rq-redis-queue.markdown b/content/pages/04-web-development/29-rq-redis-queue.markdown index d984e121a..b9fe01a39 100644 --- a/content/pages/04-web-development/29-rq-redis-queue.markdown +++ b/content/pages/04-web-development/29-rq-redis-queue.markdown @@ -19,7 +19,7 @@ track of tasks in the queue that need to be executed. ### RQ resources * [Asynchronous Tasks in Python with Redis Queue](https://www.twilio.com/blog/asynchronous-tasks-in-python-with-redis-queue) is a quickstart-style tutorial that shows how to use RQ to fetch data - from the + from the [Mars Rover web API](https://data.nasa.gov/Space-Science/Mars-Rover-Photos-API/929k-jizu) and process URLs for each of the photos taken by NASA's Mars rover. There is also a follow-up post on diff --git a/content/pages/04-web-development/31-static-site-generator.markdown b/content/pages/04-web-development/31-static-site-generator.markdown index 7b07bc047..d7f3c0983 100644 --- a/content/pages/04-web-development/31-static-site-generator.markdown +++ b/content/pages/04-web-development/31-static-site-generator.markdown @@ -206,12 +206,6 @@ point a domain name to your site as well as provide HTTPS support. These guides walk through various ways of handling the static site deployment. -* [Static site hosting with S3 and Cloudflare](https://wsvincent.com/static-site-hosting-with-s3-and-cloudflare/) - shows how to set up an S3 bucket with Cloudflare in front as a CDN that - serves the content with HTTPS. You should be able to accomplish roughly - the same situation with Amazon Cloudfront, but as a Cloudflare user I - like their service for these static site configurations. - * Google Cloud provides a tutorial on how to use them to [host your static site](https://cloud.google.com/storage/docs/hosting-static-website). Note that you cannot currently use HTTPS on Google Storage servers, which is a @@ -221,7 +215,7 @@ deployment. making static site deployments and redeployments to Amazon Web Services easier. -* [Deploying a Static Blog with Continuous Integration](https://www.loxodrome.io/post/hugo-on-ci/) +* [Deploying a Static Blog with Continuous Integration](https://www.jameslmilner.com/post/hugo-on-ci/) uses a Hugo (a Golang-based static site generator) generated site as an example but the instructions can easily be used to deploy a Python-based static site generator output as well. diff --git a/content/pages/04-web-development/32-pelican.markdown b/content/pages/04-web-development/32-pelican.markdown index 064a3adf9..c1b0c0810 100644 --- a/content/pages/04-web-development/32-pelican.markdown +++ b/content/pages/04-web-development/32-pelican.markdown @@ -76,10 +76,6 @@ to hosting services such as Amazon S3 and GitHub Pages. take in new input markup formats, modify the generator process and add handy features such as a custom table of contents. -* [Pelican Sitemap and Pagination](http://www.vcheng.org/2014/02/22/pelican-sitemap-pagination/) - explains how to generate a `sitemap.xml` file for your static site that - includes all pages instead of just auto-included top-level pages. - * [Getting started with Pelican and GitHub pages](http://www.mattmakai.com/introduction-to-pelican.html) is a tutorial I wrote to use the Full Stack Python source code to create and deploy your first static site. diff --git a/content/pages/04-web-development/35-testing.markdown b/content/pages/04-web-development/35-testing.markdown index 1a5e2912e..7594935d2 100644 --- a/content/pages/04-web-development/35-testing.markdown +++ b/content/pages/04-web-development/35-testing.markdown @@ -176,10 +176,10 @@ use mocks in your test cases. provides a whole code example based on a blog project that shows how to use `mock` when testing. -* [Python Mocking 101: Fake It Before You Make It](https://blog.fugue.co/2016-02-11-python-mocking-101.html) +* [Python Mocking 101: Fake It Before You Make It](https://www.fugue.co/blog/2016-02-11-python-mocking-101) explains what mocking is and is not, and shows how to use the `patch` function to accomplish it in your project. - [Revisiting Unit Testing and Mocking in Python](https://blog.fugue.co/2017-07-18-revisiting-unit-testing-and-mocking-in-python.html) + [Revisiting Unit Testing and Mocking in Python](https://www.fugue.co/blog/2017-07-18-revisiting-unit-testing-and-mocking-in-python.html) is a follow-up post that expands upon using the `patch` function along with dependency injection. @@ -191,12 +191,6 @@ use mocks in your test cases. examines when mocks are necessary and when they are not as useful so you can avoid them in your test cases. -* [Mocking Redis & Expiration in Python](http://malexandre.fr/2017/10/08/mocking-redis--expiration-in-python/) - is a specific scenario where you would want to test your - [Redis](/redis.html)-dependent code but prefer to mock it rather than - ensure an installation and connection are present whenever you run - your tests. - * [Better tests for Redis integrations with redislite](https://www.obeythetestinggoat.com/better-tests-for-redis-integrations-with-redislite.html) is a great example of how using the right mocking library can clean up existing hacky testing code and make it more straightforward for diff --git a/content/pages/04-web-development/37-integration-testing.markdown b/content/pages/04-web-development/37-integration-testing.markdown index 72d844b82..8594b4492 100644 --- a/content/pages/04-web-development/37-integration-testing.markdown +++ b/content/pages/04-web-development/37-integration-testing.markdown @@ -36,9 +36,6 @@ during development so they can be addressed immediately. gives an example of a system that needs integration tests and shows how context managers can be used to address the problem. -* Pytest has a page on [integration good practices](http://doc.pytest.org/en/latest/goodpractices.html) - that you'll likely want to follow when testing your application. - * [Integration testing, or how to sleep well at night](http://enterprisecraftsmanship.com/2015/07/13/integration-testing-or-how-to-sleep-well-at-nights/) explains what integration tests are and gives an example. The example is coded in Java but still relevant when you're learning about integration diff --git a/content/pages/04-web-development/38-debugging.markdown b/content/pages/04-web-development/38-debugging.markdown index e2936c6ae..8cf576d27 100644 --- a/content/pages/04-web-development/38-debugging.markdown +++ b/content/pages/04-web-development/38-debugging.markdown @@ -137,10 +137,6 @@ give solid programming language-agnostic debugging advice. * [The art of debugging](https://remysharp.com/2015/10/14/the-art-of-debugging) provides a whirlwind overview for how to fix issues in your code. -* [How to debug JavaScript errors](https://rollbar.com/guides/how-to-debug-javascript/) - introduces some key debugging tools such as source maps that make - identifying errors on the client side much easier during development. - * [Linux debugging tools you'll love](https://jvns.ca/debugging-zine.pdf) is an awesome comic that covers the Linux ecosystem for debugging. diff --git a/content/pages/04-web-development/41-https.markdown b/content/pages/04-web-development/41-https.markdown index ffe55e7c9..3bbb8239a 100644 --- a/content/pages/04-web-development/41-https.markdown +++ b/content/pages/04-web-development/41-https.markdown @@ -64,7 +64,3 @@ client web browser. cover the high-level information on the latest approved version of Transport Security Layer (TLS) 1.3. -* [How https works](https://www.sudhakar.online/programming/2015/08/09/https.html) - is a fun cartoon illustration that demonstrates the basic concepts of - a secure HTTP connection. - diff --git a/content/pages/04-web-development/42-websockets.markdown b/content/pages/04-web-development/42-websockets.markdown index fb8293c46..2a902f79c 100644 --- a/content/pages/04-web-development/42-websockets.markdown +++ b/content/pages/04-web-development/42-websockets.markdown @@ -7,10 +7,10 @@ sidebartitle: WebSockets meta: WebSockets are a protocol for full-duplex web communications. Learn about WebSockets on Full Stack Python. -A WebSocket is a [standard protocol](http://tools.ietf.org/html/rfc6455) for -two-way data transfer between a client and server. The WebSockets protocol -does not run over HTTP, instead it is a separate implementation on top of -[TCP](http://en.wikipedia.org/wiki/Transmission_Control_Protocol). +A WebSocket is a [standard protocol](https://datatracker.ietf.org/doc/html/rfc6455) +for two-way data transfer between a client and server. The WebSockets +protocol does not run over HTTP, instead it is a separate implementation +on top of [TCP](http://en.wikipedia.org/wiki/Transmission_Control_Protocol). ## Why use WebSockets? @@ -173,12 +173,6 @@ own project. that demos sending server generated events as well as input from users via a text box input on a form. -* The [realtime codenames game](https://github.com/joshporter1/codenames) - source code is a full-featured example for using WebSockets via - Flask-SocketIO. There is also a - [multi-part tutorial](https://secdevops.ai/weekend-project-part-1-creating-a-real-time-web-based-application-using-flask-vue-and-socket-b71c73f37df7) - that walks through the code. - * The [python-websockets-example](https://github.com/mattmakai/python-websockets-example) contains code to create a simple web application that provides WebSockets @@ -192,13 +186,6 @@ own project. Flask web app implementation that allows the audience to interact with WebSockets as I built out the application. -* [Creating a Real-time Web-based Application using Flask, Vue, and Socket.io: part 1](https://secdevops.ai/weekend-project-part-1-creating-a-real-time-web-based-application-using-flask-vue-and-socket-b71c73f37df7), - [part 2](https://secdevops.ai/weekend-project-part-2-turning-flask-into-a-real-time-websocket-server-using-flask-socketio-ab6b45f1d896) - and - [part 3](https://secdevops.ai/weekend-project-part-3-centralizing-state-management-with-vuex-5f4387ebc144) - are a complete front-to-backend WebSockets, Python and JavaScript front - end framework example with open source code. - * [Real-time in Python](http://mrjoes.github.io/2013/06/21/python-realtime.html) provides Python-specific context for how the server push updates were implemented in the past and how Python's tools have evolved to perform @@ -206,9 +193,7 @@ own project. * [websockets](https://github.com/aaugustin/websockets) is a WebSockets implementation for Python 3.3+ written with the - [asyncio](https://docs.python.org/3.4/library/asyncio.html) module (or with - [Tulip](https://code.google.com/p/tulip/) if you're working with - Python 3.3). + [asyncio](https://docs.python.org/3.9/library/asyncio.html) module. * [Speeding up Websockets 60X](https://www.willmcgugan.com/blog/tech/post/speeding-up-websockets-60x/) is a cool experiment in coding loops different ways to eek out more @@ -217,10 +202,6 @@ own project. of how tweaking and tuning can produce outsized returns in some applications. -* The [Choose Your Own Adventure Presentations](https://www.twilio.com/blog/2014/11/choose-your-own-adventure-presentations-with-reveal-js-python-and-websockets.html) - tutorial uses WebSockets via gevent on the server and socketio.js for - pushing vote count updates from the server to the client. - * [Adding Real Time to Django Applications](http://crossbar.io/docs/Adding-Real-Time-to-Django-Applications/) shows how to use Django and Crossbar.io to implement a publish/subscribe feature in the application. diff --git a/content/pages/04-web-development/43-webrtc.markdown b/content/pages/04-web-development/43-webrtc.markdown index 36cf18799..e34ac044a 100644 --- a/content/pages/04-web-development/43-webrtc.markdown +++ b/content/pages/04-web-development/43-webrtc.markdown @@ -26,11 +26,6 @@ browser) and server (usually a [web server](/web-servers.html)). dependencies. It allows 2 web browsers to exchange audio and video streams by using the `aiohttp` and `python-socketio` modules. -* [A real world guide to WebRTC](https://deepstreamhub.com/tutorials/protocols/webrtc-intro/) - goes through WebRTC fundamentals such as data channels, audio and video, - screen sharing and file transfers with the JavaScript code provided - for each concept. - * The [Introduction to WebRTC video series](https://www.youtube.com/watch?v=ujpIAWmK2Vo) ([part 2](https://www.youtube.com/watch?v=cw2iTgIW-uk) and @@ -38,10 +33,6 @@ browser) and server (usually a [web server](/web-servers.html)). at points but overall has a ton of good information that gives a solid overview of the technology. -* [Building a Snapchat-like app with WebRTC in the browser](https://tokbox.com/blog/building-a-snapchat-like-app-with-webrtc-in-the-browser/) - walks through the front end JavaScript for building a photo filter - application using the WebRTC browser APIs. - * [WebRTC issues and how to debug them](https://blog.codeship.com/webrtc-issues-and-how-to-debug-them/) explains the various ways that implementations can go wrong and where to start looking when you run into errors. diff --git a/content/pages/04-web-development/44-web-apis.markdown b/content/pages/04-web-development/44-web-apis.markdown index a0e1eea19..77ce68888 100644 --- a/content/pages/04-web-development/44-web-apis.markdown +++ b/content/pages/04-web-development/44-web-apis.markdown @@ -43,7 +43,7 @@ Webhooks are important because they enable two-way communication initiation for APIs. Webhook flexibility comes in from their definition by the API user instead of the API itself. -For example, in the [Twilio API](https://www.twilio.com/api) when a text +For example, in the [Twilio API](https://www.twilio.com/docs/api) when a text message is sent to a Twilio phone number Twilio sends an HTTP POST request webhook to the URL specified by the user. The URL is defined in a text box on the number's page on Twilio as shown below. @@ -91,11 +91,6 @@ on the number's page on Twilio as shown below. provides context for why JSON-based web services are more common today than SOAP which was popular in the early 2000s. -* [API tools for every occasion](https://medium.com/@orliesaurus/api-tools-for-every-occasion-10-api-tools-released-in-2015-i-can-t-live-without-d5947d9ca9c3) - provides a list of 10 tools that are really helpful when working with APIs - that are new in 2015. - - ## APIs learning checklist 1. Learn the API concepts of machine-to-machine communication with JSON and diff --git a/content/pages/04-web-development/45-microservices.markdown b/content/pages/04-web-development/45-microservices.markdown index bdf34a8b9..0b06889e6 100644 --- a/content/pages/04-web-development/45-microservices.markdown +++ b/content/pages/04-web-development/45-microservices.markdown @@ -40,22 +40,6 @@ ease further development and deployment. This approach is called the article is one of the best in-depth explanations for what microservices are and why to consider them as an architectural pattern. -* [Why microservices?](http://dev.otto.de/2016/03/20/why-microservices/) - presents some of the advantages, such as the dramatically increased number - of deployments per day, that a well-done microservices architecture can - provide in the right situation. Many organizational environments won't - allow this level of flexibility but if yours is one that will, it's worth - considering these points. - -* [On monoliths and microservices](http://dev.otto.de/2015/09/30/on-monoliths-and-microservices/) - provides some advice on using microservices in a fairly early stage of - a software project's lifecycle. - -* [Why Microservices?](https://dev.otto.de/2016/03/20/why-microservices/) - presents advantages microservices can bring to an existing monolithic - application where it is clear what needs to be broken down into smaller - components to make it easier to iterate and maintain. - * [Developing a RESTful microservice in Python](http://www.skybert.net/python/developing-a-restful-micro-service-in-python/) is a good story of how an aging Java project was replaced with a microservice built with Python and Flask. diff --git a/content/pages/04-web-development/46-webhooks.markdown b/content/pages/04-web-development/46-webhooks.markdown index 01d76fe43..6583c526d 100644 --- a/content/pages/04-web-development/46-webhooks.markdown +++ b/content/pages/04-web-development/46-webhooks.markdown @@ -33,19 +33,12 @@ otherwise independent web applications. explanation of this concept that also contains some basic security considerations when using them. -* [How to Listen for Webhooks with Python](https://blog.bearer.sh/consume-webhooks-with-python/) - has code examples in both [Flask](/flask.html) and [Django](/django.html) - for how to receive an HTTP POST webhook request, as well as how to test - it locally with Ngrok. - * [Webhooks for Beginners - Full Course](https://www.youtube.com/watch?v=41NOoEz3Tzc) is an entire free video course that shows both how to use and implement webhooks into applications. * [Should you build a webhooks API?](https://brandur.org/webhooks) -* [Webhooks do’s and dont’s: what we learned after integrating +100 APIs](https://restful.io/webhooks-dos-and-dont-s-what-we-learned-after-integrating-100-apis-d567405a3671) - * [Why Are Webhooks Better Than Serverless Extensibility?](https://developer.okta.com/blog/2017/10/11/why-are-webhooks-better-than-serverless-extensibility) * [Webhooks Provide an Efficient Alternative to API Polling](https://thenewstack.io/wonderful-world-webhooks/) diff --git a/content/pages/04-web-development/48-api-creation.markdown b/content/pages/04-web-development/48-api-creation.markdown index da38541c2..aa49c5aa1 100644 --- a/content/pages/04-web-development/48-api-creation.markdown +++ b/content/pages/04-web-development/48-api-creation.markdown @@ -26,9 +26,6 @@ applications through machine-to-machine communication. [own GitHub organization](https://github.com/flask-restful/flask-restful) so engineers from outside the company could be core contributors. -* [Flask API](http://www.flaskapi.org/) is another common library for - exposing APIs from Flask web applications. - * [Sandman](http://www.github.com/jeffknupp/sandman) is a widely used tool to automatically generate a RESTful API service from a legacy database without writing a line of code (though it's easily extensible through code). @@ -133,12 +130,8 @@ equivalent of browser testing in the web application world. compatibility and a whole slew of other great advice for developers and API designers. -* [Self-descriptive, isn't. Don't assume anything.](http://www.bizcoder.com/self-descriptive-isn-t-don-t-assume-anything) - is an appeal that metadata makes a difference in whether APIs are descriptive - or not. - * [Designing the Artsy API](http://artsy.github.io/blog/2014/09/12/designing-the-public-artsy-api/) - has their recommendations list for building an API based on their recent + has their recommendations list for building an API based on their experiences. * Hacker News had a discussion on @@ -154,10 +147,6 @@ equivalent of browser testing in the web application world. will use your API, as well as what the documentation for endpoints and other important pieces should look like. -* [1-to-1 Relationships and Subresources in REST APIs](http://developers.lyst.com/2015/02/20/1-to-1-relationships-and-subresources-in-rest-apis/) - tells the story of design decisions that were made during an API's creation - and why those choices were made. - * [How many status codes does your API need?](https://blogs.dropbox.com/developers/2015/04/how-many-http-status-codes-should-your-api-use/) gives an answer from a Dropbox API developer as to their decision making process. @@ -219,10 +208,6 @@ equivalent of browser testing in the web application world. [Serpy](https://github.com/clarkduvall/serpy) and [wrote a blog post with the results of its performance](https://engineering.betterworks.com/2015/09/04/ditching-django-rest-framework-serializers-for-serpy/). -* [Designing a Web API](http://restlet.com/company/blog/2015/03/16/api-design-designing-a-web-api/) - gives a detailed walkthrough of concepts and design decisions you need - to make when building an API. - * Microsoft's [REST API Guidelines](https://github.com/Microsoft/api-guidelines/blob/master/Guidelines.md) are a detailed set of considerations for when you are building your own diff --git a/content/pages/04-web-development/50-django-rest-framework-drf.markdown b/content/pages/04-web-development/50-django-rest-framework-drf.markdown index 5824966bc..8e37986e0 100644 --- a/content/pages/04-web-development/50-django-rest-framework-drf.markdown +++ b/content/pages/04-web-development/50-django-rest-framework-drf.markdown @@ -60,10 +60,6 @@ typically abbreviated "DRF", is a Python library for building web * [Optimizing slow Django REST Framework performance](https://ses4j.github.io/2015/11/23/optimizing-slow-django-rest-framework-performance/) -* [TLT: Serializing Authenticated User Data With Django REST Framework](http://gregblogs.com/tlt-serializing-authenticated-user-data-with-django-rest-framework/) - -* [Building an API with Django REST Framework and Class-Based Views](https://codeburst.io/building-an-api-with-django-rest-framework-and-class-based-views-75b369b30396) - * [Simple Nested API Using Django REST Framework](https://blog.apptension.com/2017/09/13/rest-api-using-django-rest-framework/) * [Building APIs with Django and Django Rest Framework](https://books.agiliq.com/projects/django-api-polls-tutorial/en/latest/) diff --git a/content/pages/04-web-development/51-api-integration.markdown b/content/pages/04-web-development/51-api-integration.markdown index 09c6e8099..ee646bfa9 100644 --- a/content/pages/04-web-development/51-api-integration.markdown +++ b/content/pages/04-web-development/51-api-integration.markdown @@ -63,9 +63,6 @@ across many implementation areas. is a nice tutorial for easily re-executing failed HTTP requests with the Requests library. -* My DjangoCon 2013 talk dove into - "[Making Django Play Nice With Third Party Services](http://www.youtube.com/watch?v=iGP8DQIqxXs)." - * If you're looking for a fun project that uses two web APIs within a Django application, try out this tutorial to [Build your own Pokédex with Django, MMS and PokéAPI](https://www.twilio.com/blog/2014/11/build-your-own-pokedex-with-django-mms-and-pokeapi.html). diff --git a/content/pages/04-web-development/52-twilio.markdown b/content/pages/04-web-development/52-twilio.markdown index 9ba51f456..e038e3907 100644 --- a/content/pages/04-web-development/52-twilio.markdown +++ b/content/pages/04-web-development/52-twilio.markdown @@ -77,10 +77,6 @@ for fellow developers. [Twilio SMS API via some Python code](https://www.twilio.com/docs/quickstart/python/sms) to send a text message with the results. -* IBM's Bluemix blog contains a nice tutorial on building an - [IoT Python app with a Raspberry Pi and Bluemix](https://developer.ibm.com/bluemix/2015/04/02/tutorial-using-a-raspberry-pi-python-iot-twilio-bluemix/) - that uses Twilio to interact with the Raspberry Pi. - * The [Python tag on the Twilio blog](https://www.twilio.com/blog/tag/python) provides walkthroughs for [Django](https://www.twilio.com/blog/2015/12/city-chat-with-python-django-and-twilio-ip-messaging.html), diff --git a/content/pages/04-web-development/53-stripe.markdown b/content/pages/04-web-development/53-stripe.markdown index aee4f6f98..82aba71f0 100644 --- a/content/pages/04-web-development/53-stripe.markdown +++ b/content/pages/04-web-development/53-stripe.markdown @@ -23,9 +23,6 @@ for processing payments. the subscription data in the [Django ORM](/django-orm.html) and create a pricing page. -* [Switching from Braintree to Stripe](https://www.deekit.com/braintree-to-stripe/) - covers one development team's experience with moving payment providers. - * [Dirt Cheap Recurring Payments with Stripe and AWS Lambda](http://normal-extensions.com/2017/05/05/simple-recurring/) explains how to use the Stripe API with [AWS Lambda](/aws-lambda.html) to handle recurring payments instead of using a more expensive service diff --git a/content/pages/04-web-development/56-web-app-security.markdown b/content/pages/04-web-development/56-web-app-security.markdown index 2b1ee70f9..992091dbf 100644 --- a/content/pages/04-web-development/56-web-app-security.markdown +++ b/content/pages/04-web-development/56-web-app-security.markdown @@ -14,8 +14,6 @@ request forgery and usage of public-private keypairs. ### Security tools -* [Bro](http://www.bro.org/) is a network security and traffic monitor. - * [lynis](https://cisofy.com/lynis/) ([source code](https://github.com/CISOfy/lynis)) is a security audit tool that can run as a shell script on a Linux system to find out @@ -112,7 +110,7 @@ resources can also give you a good overview of how HTTPS works. learn about web browser internals, session attacks, fingerprinting, HTTPS and many other fundamental topics. -* [The SaaS CTO Security Checklist](https://cto-security-checklist.sqreen.io/) +* [The SaaS CTO Security Checklist Redux](https://www.goldfiglabs.com/guide/saas-cto-security-checklist/) is an awesome list of steps for securing your infrastructure and employees as well as what stage and size company it is recommended that you put those procedures in place. @@ -132,11 +130,6 @@ resources can also give you a good overview of how HTTPS works. * The [/r/netsec](http://www.reddit.com/r/netsec/) subreddit is one place to go to learn more about network and application security. -* [Hacking Tools Repository](http://gexos.github.io/Hacking-Tools-Repository/) - is a great list of password cracking, scanning, sniffing and other security - penetration testing tools. - - * The EFF has a well written overview on [what makes a good security audit](https://www.eff.org/deeplinks/2014/11/what-makes-good-security-audit). It's broad but contains some of their behind the scenes thinking on important considerations with security audits. @@ -160,10 +153,6 @@ resources can also give you a good overview of how HTTPS works. important technique to use to keep your database passwords and other secrets more secure if the hashed strings are leaked. -* [An in-depth analysis of SSH attacks on Amazon EC2](http://getprismatic.com/story/1409447605839) - shows how important it is to secure your web servers, especially when they are - hosted in IP address ranges that are commonly scanned by malicious actors. - * [Cloud Security Auditing: Challenges and Emerging Approaches](http://www.infoq.com/articles/cloud-security-auditing-challenges-and-emerging-approaches) is a high-level overview of some of security auditing problems that come with cloud deployments. diff --git a/content/pages/05-deployment/00-deployment.markdown b/content/pages/05-deployment/00-deployment.markdown index a7080da8a..48ad4e8c1 100644 --- a/content/pages/05-deployment/00-deployment.markdown +++ b/content/pages/05-deployment/00-deployment.markdown @@ -51,7 +51,7 @@ guide as they are considered advanced deployment techniques. * [teletraan](https://github.com/pinterest/teletraan) is the deploy system used by the development teams at Pinterest, a huge Python shop! -* [pants](https://www.pantsbuild.org/index.html) is a build system originally +* [pants](https://www.pantsbuild.org/) is a build system originally created at Twitter and now split out as its own sustainable open source project. @@ -132,14 +132,6 @@ guide as they are considered advanced deployment techniques. is an awesome in-depth read covering topics ranging from git branching to database migrations. -* In [this free video by Neal Ford](http://player.oreilly.com/videos/9781491908181?toc_id=210188), - he talks about engineering practices for continuous delivery. He explains - the difference between - [continuous integration](/continuous-integration.html), - continuous deployment and continuous delivery. Highly recommended for an - overview of deployment concepts and as an introduction to the other videos - on those subjects in that series. - * [TestDriven.io](https://testdriven.io/) shows how to deploy a [microservices](/microservices.html) architecture that uses [Docker](/docker.html), [Flask](/flask.html), and React with diff --git a/content/pages/05-deployment/02-servers.markdown b/content/pages/05-deployment/02-servers.markdown index 90bb07951..f66f68977 100644 --- a/content/pages/05-deployment/02-servers.markdown +++ b/content/pages/05-deployment/02-servers.markdown @@ -128,8 +128,8 @@ provides a unified API for many cloud service providers. * [Amazon Web Services has official documentation](http://aws.amazon.com/python/) for running Python web applications. -* [boto](https://github.com/boto/boto) is an extensive and well-tested -Python library for working with Amazon Web Services. +* [boto3](https://github.com/boto/boto3) is an extensive and well-tested + Python library for working with Amazon Web Services. * [Poseidon](https://github.com/changhiskhan/poseidon) is a Python commandline interface for managing Digital Ocean droplets (servers). diff --git a/content/pages/05-deployment/03-static-content.markdown b/content/pages/05-deployment/03-static-content.markdown index 8c6e88262..f5a118f54 100644 --- a/content/pages/05-deployment/03-static-content.markdown +++ b/content/pages/05-deployment/03-static-content.markdown @@ -21,9 +21,9 @@ Django framework calls these two categories *assets* and *media*. ## Content delivery networks A content delivery network (CDN) is a third party that stores and serves static files. [Amazon CloudFront](http://aws.amazon.com/cloudfront/), -[Akamai](http://www.akamai.com/), and -[Rackspace Cloud Files](http://www.rackspace.com/cloud/public/files/) -are examples of CDNs. The purpose of a CDN is to remove the load of static +[CloudFlare](https://www.cloudflare.com/) and [Fastly](https://www.fastly.com/), +are examples of CDN services. The purpose of +a CDN is to remove the load of static file requests from web servers that are handling dynamic web content. For example, if you have an nginx server that handles both static files and acts as a front for a Green Unicorn WSGI server on a 512 megabyte diff --git a/content/pages/05-deployment/04-cdns.markdown b/content/pages/05-deployment/04-cdns.markdown index 027f064d6..bd4e516fe 100644 --- a/content/pages/05-deployment/04-cdns.markdown +++ b/content/pages/05-deployment/04-cdns.markdown @@ -12,9 +12,9 @@ servers to improve web app loading speed. ### CDN resources -* [Mastering HTTP Caching](https://blog.fortrabbit.com/mastering-http-caching) - is a fantastic post that goes into great technical detail on how CDNs and - caching work. +* [The 5 hour CDN](https://fly.io/blog/the-5-hour-content-delivery-network/) + explains the basics of what CDNs are and how they are a combination of + many standard web server components, but used globally and at scale. * [MaxCDN vs CloudFlare vs Amazon CloudFront vs Akamai Edge vs Fastly](https://www.codeinwp.com/blog/maxcdn-vs-cloudflare-vs-cloudfront-vs-akamai-edge-vs-fastly/) compares and contrasts the most popular CDN services based on features, @@ -29,10 +29,6 @@ servers to improve web app loading speed. [django-storages](https://django-storages.readthedocs.io/en/latest/) library to deploy static assets for a [Django](/django.html) application to a CDN. -* [Building your own CDN for Fun and Profit](https://pasztor.at/blog/building-your-own-cdn) - is a great high-level overview of how CDNs work and shows you how to - create your own, albeit simplified CDN. - * [Do not let your CDN betray you: Use Subresource Integrity](https://hacks.mozilla.org/2015/09/subresource-integrity-in-firefox-43/) describes the security implications for CDNs with unexpectedly modified content and how Subresource Integrity in modern web browsers can mitigate diff --git a/content/pages/05-deployment/09-paas.markdown b/content/pages/05-deployment/09-paas.markdown index 9416e971f..c89a37439 100644 --- a/content/pages/05-deployment/09-paas.markdown +++ b/content/pages/05-deployment/09-paas.markdown @@ -69,11 +69,6 @@ of controlling and modifying the project for your own applications, but prevents you from offloading the responsibility of keeping servers running to someone else. -* [Kel](http://www.kelproject.com/) uses Kubernetes as a foundation - for a custom self-hosted PaaS. Note that it was created by Eldarion, - which had one of the first Python-specific PaaS offerings on the - market around the time that Heroku was launched. - * [Dokku](http://dokku.viewdocs.io/dokku/) builds on Docker and has hooks for plugins to extend the small core of the project and customize deployments for your applications. @@ -82,7 +77,6 @@ running to someone else. designed to run on top of AWS services. - ## Platform-as-a-service resources * [The differences between IaaS, PaaS and SaaS](https://www.engineyard.com/blog/the-differences-between-iaas-paas-and-saas-and-when-to-use-each) explains the abstract layer differences among "X-as-a-service" offering @@ -156,11 +150,6 @@ running to someone else. on Google Cloud and posits what they may be paying to run their service. -* [PaaS (false) economics](https://blog.drie.co/paas-false-economics-13f72d87b485) - gives some quick back-of-the-envelope calculations on why running your - applications on a PaaS is obviously going to appear more expensive if you - do not take the cost of your own software engineers into the equation. - * Two blog posts on using AWS Autoscaling in [Automatic replacement of Autoscaling nodes with equivalent spot instances](https://mcristi.wordpress.com/2016/04/21/my-approach-at-making-aws-ec2-affordable-automatic-replacement-of-autoscaling-nodes-with-equivalent-spot-instances/) and [Autoscaling nodes: seeing it in action](https://mcristi.wordpress.com/2016/04/27/automatic-replacement-of-autoscaling-nodes-with-equivalent-spot-instances-seeing-it-in-action/) diff --git a/content/pages/05-deployment/10-heroku.markdown b/content/pages/05-deployment/10-heroku.markdown index 63aad7441..06c1d7bcb 100644 --- a/content/pages/05-deployment/10-heroku.markdown +++ b/content/pages/05-deployment/10-heroku.markdown @@ -21,7 +21,14 @@ easily [deploy](/deployment.html) Python applications. [MySQL](/mysql.html) to [PostgreSQL](/postgresql.html) if necessary as well as how to properly handle your settings files. -* Heroku's +* [How to deploy Django project to Heroku using Docker](https://www.accordbox.com/blog/deploy-django-project-heroku-using-docker/) + explains that although [Buildpacks](https://devcenter.heroku.com/articles/buildpacks) + are the most common way to deploy to Heroku, packaing your app in a + [Docker](/docker.html) container is also a viable approach. It walks through + the steps needed to deploy a [Django](/django.html) app in the remainder + of the article. + +* Heroku's [official Python documentation](https://devcenter.heroku.com/articles/getting-started-with-python) is fantastic and walks through deploying WSGI applications in short order. diff --git a/content/pages/05-deployment/19-nginx.markdown b/content/pages/05-deployment/19-nginx.markdown index afe620d84..3f17e6524 100644 --- a/content/pages/05-deployment/19-nginx.markdown +++ b/content/pages/05-deployment/19-nginx.markdown @@ -63,7 +63,7 @@ to make sure you are avoiding the most common security errors that plague HTTP(S) configurations. * [HTTPS with Let's Encrypt and nginx](https://botleg.com/stories/https-with-lets-encrypt-and-nginx/) - walks throough installing a free SSL certificate from Let's Encrypt + walks through installing a free SSL certificate from Let's Encrypt to secure HTTP connects to your nginx server via HTTPS. * The [Nginx Config](https://nginxconfig.io/) tool can generate strong @@ -74,9 +74,7 @@ HTTP(S) configurations. * [Strong SSL Security on Nginx](https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html) shows how to mitigate high profile SSL attacks like - [Logjam](https://weakdh.org/), - [Heartbleed](http://heartbleed.com/) - and [FREAK](https://freakattack.com/). + [Logjam](https://weakdh.org/) and [Heartbleed](http://heartbleed.com/). ## Specific Nginx resources @@ -103,7 +101,7 @@ several years. * [Test-driving web server configuration](https://gdstechnology.blog.gov.uk/2015/03/25/test-driving-web-server-configuration/) tells a good story for how to iteratively apply configuration changes, such - as routing traffic to [Piwik](http://piwik.org/) for + as routing traffic to [Matoma](https://matomo.org/) for [web analytics](/web-analytics.html), reverse proxying to backend application servers and terminately TLS connections appropriately. It is impressive to read a well-written softare development article like @@ -118,10 +116,6 @@ several years. as well as the Pagespeed module that Google released for both Nginx and the [Apache HTTP Server](/apache-http-server.html). -* [Nginx for Developers: An Introduction](http://carrot.is/coding/nginx_introduction) - provides the first steps to getting an initial Nginx configuration up and - running. - * [A faster Web server: ripping out Apache for Nginx](http://arstechnica.com/business/2011/11/a-faster-web-server-ripping-out-apache-for-nginx/) explains how Nginx can be used instead of Apache in some cases for better performance. diff --git a/content/pages/05-deployment/20-caddy.markdown b/content/pages/05-deployment/20-caddy.markdown index 1b27e6d11..56ca43167 100644 --- a/content/pages/05-deployment/20-caddy.markdown +++ b/content/pages/05-deployment/20-caddy.markdown @@ -15,7 +15,8 @@ and design emphasize HTTPS-everywhere along with the HTTP/2 protocol. ## How can Caddy be used with Python deployments? Caddy can be used both for testing during local development or as part of a production deployment as an HTTP server and a reverse proxy with -the [proxy directive](https://caddyserver.com/docs/proxy). +the +[reverse_proxy directive](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy).
Caddy is an implementation of the web server concept. Learn how these pieces fit together in the deployment chapter or view the table of contents for all topics.
diff --git a/content/pages/05-deployment/22-wsgi-servers.markdown b/content/pages/05-deployment/22-wsgi-servers.markdown index 35174df41..036bdbe54 100644 --- a/content/pages/05-deployment/22-wsgi-servers.markdown +++ b/content/pages/05-deployment/22-wsgi-servers.markdown @@ -177,9 +177,9 @@ The following are WSGI servers based on community recommendations. is a good read to understand basic information about various WSGI server implementations. -* A thorough and informative post for LAMP-stack hosting choices is - presented in the - "[complete single server Django stack tutorial](http://www.apreche.net/complete-single-server-django-stack-tutorial/)." +* [What is WSGI and Why Do You Need Gunicorn and Nginx in Django](https://apirobot.me/posts/what-is-wsgi-and-why-do-you-need-gunicorn-and-nginx-in-django) + explains the breakdown between a [web server](/web-servers.html) + and a WSGI server in an application deployment environment. * The Python community made a long effort to [transition from mod\_python](http://blog.dscpl.com.au/2010/05/modpython-project-soon-to-be-officially.html) diff --git a/content/pages/05-deployment/23-gunicorn.markdown b/content/pages/05-deployment/23-gunicorn.markdown index f7098a1b4..1973ab6ff 100644 --- a/content/pages/05-deployment/23-gunicorn.markdown +++ b/content/pages/05-deployment/23-gunicorn.markdown @@ -49,7 +49,7 @@ file with the following contents: It exposes the WSGI callable as a module-level variable named ``application``. For more information on this file, see - https://docs.djangoproject.com/en/1.8/howto/deployment/wsgi/ + https://docs.djangoproject.com/en/3.2/howto/deployment/wsgi/ """ import os @@ -115,11 +115,15 @@ perform the request handling. Each worker is independent of the controller. screenshots along the way with what to expect while you are configuring the deployment server. -* The [Django](https://docs.djangoproject.com/en/1.9/howto/deployment/wsgi/gunicorn/) +* The [Django](https://docs.djangoproject.com/en/3.2/howto/deployment/wsgi/gunicorn/) and [Flask](http://flask.pocoo.org/docs/latest/deploying/wsgi-standalone/) documentation each contain instructions for deploying the respective frameworks with Gunicorn. +* [Dockerizing Django with Postgres, Gunicorn, and Traefik](https://testdriven.io/blog/django-docker-traefik/) + is a more advanced tutorial with a complete project deployment + that uses Gunicorn within [Docker](/docker.html). + * [Set up Django, Nginx and Gunicorn in a Virtualenv controled by Supervisor](https://gist.github.com/Atem18/4696071) is a GitHub Gist with some great explanations for why we're setting up virtualenv and what to watch out for while you're doing the deployment. diff --git a/content/pages/05-deployment/24-uwsgi.markdown b/content/pages/05-deployment/24-uwsgi.markdown index 985d326fe..7561de3da 100644 --- a/content/pages/05-deployment/24-uwsgi.markdown +++ b/content/pages/05-deployment/24-uwsgi.markdown @@ -42,7 +42,3 @@ implementation that is typically used to run Python web applications. is awesome because it shows you how to code a quick WSGI application without using a framework then builds up an example with deploying a traditional Django web app. - -* [Deployment Notes for Pylons, Nginx, and uWSGI](http://tonylandis.com/python/deployment-howt-pylons-nginx-and-uwsgi/) - gives the code and instructions for setting up a Pylons application - with uWSGI. diff --git a/content/pages/05-deployment/27-continuous-integration.markdown b/content/pages/05-deployment/27-continuous-integration.markdown index 835f5589a..ea620e337 100644 --- a/content/pages/05-deployment/27-continuous-integration.markdown +++ b/content/pages/05-deployment/27-continuous-integration.markdown @@ -127,11 +127,6 @@ programming language agnostic. Learn more via the following resources or on uses well done drawings to show how continuous integration and delivery works for testing and managing data. -* [The real difference between CI and CD](https://fire.ci/blog/the-difference-between-ci-and-cd/) - explains what advantages CI provides, what constraints it operates under - (such as total build time) to work well, and how that is different from - the related but distinct concept of continuous delivery. - * [6 top continuous integration tools](https://opensource.com/business/15/7/six-continuous-integration-tools) gives a high level overview of six CI tools from a programming language agnostic perspective. diff --git a/content/pages/05-deployment/28-jenkins.markdown b/content/pages/05-deployment/28-jenkins.markdown index 46a711870..63c64d57c 100644 --- a/content/pages/05-deployment/28-jenkins.markdown +++ b/content/pages/05-deployment/28-jenkins.markdown @@ -35,20 +35,11 @@ used to automate building, [testing](/testing.html) and is another solid tutorial that also shows how to send email notifications as part of the build process. -* If you're running into difficulty adding an SSH key to your Jenkins system - account so you can connect to another server or Git repository - [this blog post on connecting Jenkins with Git](http://dcycleproject.org/blog/51/connecting-jenkins-and-git) - to get the steps to solve that problem. - * [Running Jenkins in Docker Containers](http://www.catosplace.net/blog/2015/02/11/running-jenkins-in-docker-containers/) is a short tutorial showing how to use the official [Jenkins container](https://registry.hub.docker.com/_/jenkins/) on the Docker hub. -* [Securing Jenkins](https://wiki.jenkins.io/display/JENKINS/Securing+Jenkins) - is the landing page for Jenkins security. If you're deploying your own - instance, you'll need to lock it down against unauthorized users. - * [Updating the GOV.UK Continuous Integration environment](https://gdstechnology.blog.gov.uk/2017/02/10/updating-the-gov-uk-continuous-integration-environment/) describes the configuration improvements one infrastructure team made to Jenkins, where they enabled @@ -77,7 +68,3 @@ used to automate building, [testing](/testing.html) and * [Automated API testing with Jenkins](https://assertible.com/blog/automated-api-testing-with-jenkins) walks through how to use Jenkins to tests your [API](/application-programming-interfaces.html) upon each deployment. - -* [Continuous Delivery with Jenkins and Rollbar](https://rollbar.com/blog/continuous-delivery-with-jenkins/) - is a tutorial on using Jenkins for continuous integration paired with - [Rollbar](/rollbar.html) for tracking deployments and errors. diff --git a/content/pages/05-deployment/32-configuration-management.markdown b/content/pages/05-deployment/32-configuration-management.markdown index 252df484e..23f4b07ba 100644 --- a/content/pages/05-deployment/32-configuration-management.markdown +++ b/content/pages/05-deployment/32-configuration-management.markdown @@ -32,10 +32,6 @@ operations, such as querying the database from the Django manage.py shell. is an openly biased but detailed post on why to choose SaltStack over Ansible in certain situations. -* [Ansible vs. Shell Scripts](https://valdhaus.co/writings/ansible-vs-shell-scripts/) - provides some perspective on why using a configuration management tool is a - better choice than venerable but brittle shell scripts. - * [Ansible vs. Chef](http://tjheeta.github.io/2015/04/15/ansible-vs-chef/) is a comparsion of Ansible with the Chef configuration management tool. @@ -59,17 +55,6 @@ management and application deployment tool built in Python. is a fantastically detailed introduction on using Ansible to set up servers. -* [Ansible Text Message Notifications with Twilio SMS](https://www.twilio.com/blog/2014/05/ansible-text-messages-notifications-with-twilio-sms.html) - is my blog post with a detailed example for using the Twilio module in - core Ansible 1.6+. - -* [Python for Configuration Management with Ansible slides](http://www.insom.me.uk/post/pycon-talk.html) -from PyCon UK 2013 - -* [First Steps with Ansible](http://labs.qandidate.com/blog/2013/11/15/first-steps-with-ansible/) - -* [Red Badger on Ansible](http://red-badger.com/blog/2013/06/29/ansible/) - * [Getting Started with Ansible](http://lowendbox.com/blog/getting-started-with-ansible/) * [An introduction to Ansible](https://davidwinter.me/introduction-to-ansible/) @@ -89,8 +74,6 @@ from PyCon UK 2013 * [Idempotence, convergence, and other silly fancy words we often use](https://groups.google.com/forum/#!msg/Ansible-project/WpRblldA2PQ/lYDpFjBXDlsJ) -* [Testing with Jenkins, Docker and Ansible](http://blog.mist.io/post/82383668190/move-fast-and-dont-break-things-testing-with) - ## Application dependencies learning checklist 1. Learn about configuration management in the context of deployment diff --git a/content/pages/05-deployment/33-ansible.markdown b/content/pages/05-deployment/33-ansible.markdown index a64409f26..bbdff142a 100644 --- a/content/pages/05-deployment/33-ansible.markdown +++ b/content/pages/05-deployment/33-ansible.markdown @@ -38,9 +38,6 @@ be able to structure your playbooks: configures macOS with various applications and developer tools such as [Docker](/docker.html), Homebrew and [Sublime Text](/sublime-text.html). -* [ansible-nginx-haproxy-elasticsearch](https://github.com/gp187/ansible-nginx-haproxy-elasticsearch) - sets up a server with [Nginx](/nginx.html), HAProxy and ElasticSearch. - ### Specific Ansible topics * [An Ansible2 Tutorial](https://serversforhackers.com/c/an-ansible2-tutorial) @@ -63,10 +60,6 @@ be able to structure your playbooks: * [DevOps from Scratch, Part 1: Vagrant & Ansible](https://www.kevinlondon.com/2016/09/19/devops-from-scratch-pt-1.html) -* [Ansible: Post-Install Setup](https://valdhaus.co/writings/ansible-post-install/) - -* [How To Use Vault to Protect Sensitive Ansible Data on Ubuntu 16.04](https://www.digitalocean.com/community/tutorials/how-to-use-vault-to-protect-sensitive-ansible-data-on-ubuntu-16-04) - * [How to use Ansible Variables and Vaults](https://www.expressvpn.com/blog/ansible-variables-vaults/) * [CI for Ansible playbooks which require Ansible Vault protected variables](https://www.jeffgeerling.com/blog/2017/ci-ansible-playbooks-which-require-ansible-vault-protected-variables) diff --git a/content/pages/05-deployment/36-docker.markdown b/content/pages/05-deployment/36-docker.markdown index 8017a95da..f133b70f0 100644 --- a/content/pages/05-deployment/36-docker.markdown +++ b/content/pages/05-deployment/36-docker.markdown @@ -48,20 +48,10 @@ on Amazon Web Services, Google Compute Engine, Linode, Rackspace or elsewhere. repository and tutorial that shows you how to recreate a simplified version of Docker to better understand what it's doing under the hood. -* [Andrew Baker](https://github.com/atbaker) presented a fantastic tutorial - at [PyOhio](http://andrewtorkbaker.com/pyohio-docker-101-tutorial) on - [beginner and advanced Docker usage](https://github.com/atbaker/docker-tutorial). - Andrew also wrote the article - [what containers can do for you](http://radar.oreilly.com/2015/01/what-containers-can-do-for-you.html). - * [Docker curriculum](http://prakhar.me/docker-curriculum/) is a detailed tutorial created by a developer to show the exact steps for deploying an application that relies on [Elasticsearch](https://www.elastic.co/). -* [How To Install and Use Docker on Ubuntu 16.04](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-16-04) - provides a walkthrough for Ubuntu 16.04 for installing and beginning to - use Docker for development. - * [It Really is the Future](http://blog.circleci.com/it-really-is-the-future/) discusses Docker and containers in the context of whether it's all just a bunch of hype or if this is a real trend for infrastructure automation. @@ -87,10 +77,6 @@ on Amazon Web Services, Google Compute Engine, Linode, Rackspace or elsewhere. is a short tutorial for creating a Docker container with a specific configuration. -* [10 things to avoid in Docker containers](http://developerblog.redhat.com/2016/02/24/10-things-to-avoid-in-docker-containers/) - provides a lot of "don'ts" that you'll want to consider before bumping - up against the limitations of how containers should be used. - * [Docker Internals](http://docker-saigon.github.io/post/Docker-Internals/) presents Linux containers and how Docker uses them as its base for how the project works. This article is a great way to bridge what you know about Docker with a more diff --git a/content/pages/05-deployment/37-kubernetes.markdown b/content/pages/05-deployment/37-kubernetes.markdown index 3e029b97d..96e69ef62 100644 --- a/content/pages/05-deployment/37-kubernetes.markdown +++ b/content/pages/05-deployment/37-kubernetes.markdown @@ -30,11 +30,11 @@ applications. * [Kompose](http://kompose.io/index) ([source code](https://github.com/kubernetes/kompose)) - translate Docker Compose files into Kubernetes configuration resources. + translates Docker Compose files into Kubernetes configuration resources. -* [skaffold](https://skaffold.dev/). [Using Kubernetes for local development](https://nemethgergely.com/using-kubernetes-for-local-development/index.html) - is a good starting place for more information on getting started with - Skaffold. +* [skaffold](https://skaffold.dev/) + ([source code](https://github.com/GoogleContainerTools/skaffold)) makes + it easier to develop locally with Kubernetes. * [kubethanos](https://github.com/berkay-dincer/kubethanos) is a tool to kill half of your Kubernetes pods at random, to test the resilience of your @@ -88,7 +88,7 @@ applications. cost-benefit analysis to make sure the tool's scability, reliability and related functionality will outweigh the downsides. -* [How Zolando manages 140+ Kubernetes clusters](https://srcco.de/posts/how-zalando-manages-140-kubernetes-clusters.html) +* [How Zalando manages 140+ Kubernetes clusters](https://srcco.de/posts/how-zalando-manages-140-kubernetes-clusters.html) covers the architecture, monitoring and workflow of a team that has to run a decent number of clusters for their development teams. diff --git a/content/pages/05-deployment/38-serverless.markdown b/content/pages/05-deployment/38-serverless.markdown index ad05af5f1..a5236c280 100644 --- a/content/pages/05-deployment/38-serverless.markdown +++ b/content/pages/05-deployment/38-serverless.markdown @@ -45,9 +45,8 @@ These implementations are under significant active development and not all of them have Python support. * [AWS Lambda](/aws-lambda.html) is the current leader among serverless - compute implementations. It has support for both - [Python 2.7](/blog/aws-lambda-python-2-7.html) and - [Python 3.6/3.7](/blog/aws-lambda-python-3-6.html). + compute implementations. It has support for + [Python 3.x](/blog/aws-lambda-python-3-6.html). * Azure Functions has second-class citizen support for Python. It's supposed to be possible but @@ -59,14 +58,11 @@ and not all of them have Python support. [Apache OpenWhisk](https://github.com/openwhisk/openwhisk) open source project. -* [Google Cloud Functions](/google-cloud-functions.html) currently - only supports JavaScript code execution. +* [Google Cloud Functions](/google-cloud-functions.html) has + [native Python 3.x runtimes](https://cloud.google.com/functions/docs/concepts/python-runtime). -* Webtask.io also only supports JavaScript but there is a cool - *prototype* project named [webtask-pytask](https://github.com/tehsis/webtask-pytask) - to run Python code in the browser via webtask. This demo is definitely not - for production code use but awesome to see what the programming community - can put together using existing code and services. +* [Webtask.io](https://webtask.io/) started as a JavaScript service but + now also has a Python runtime as well. ### Serverless frameworks @@ -83,8 +79,7 @@ include: which is a useful but generically-named library that focuses on deployment and operations for serverless applications. -* [Zappa](https://www.zappa.io/) - ([source code](https://github.com/Miserlou/Zappa)) +* [Zappa](https://github.com/Miserlou/Zappa) provides code and tools to make it much easier to build on AWS Lambda and AWS API Gateway than rolling your own on the bare services. @@ -92,8 +87,6 @@ include: ([source code](https://github.com/aws/chalice)) is built by the AWS team specifically for Python applications. -* [Apex](http://apex.run/) ([source code](https://github.com/apex/apex)) - ### General serverless resources Serverless concepts and implementations are still in their early @@ -189,37 +182,18 @@ have varying degrees of support for Python. AWS Lambda has production-ready support for Python 2 and 3.7, while Azure and Google Cloud have "beta" support with unclear production-worthiness. The following resources are some comparison articles to help you in your decision-making -process for which platform to learn. - -* [Serverless at scale](https://blog.binaris.com/serverless-at-scale/) - compares the "Big 3" AWS, Azure and Google Cloud in serverless performance. - The author provides some nice data around average response times and - outliers. - -* [Serverless hosting comparison](https://headmelted.com/serverless-showdown-4a771ca561d2) - is a broad overview of documentation, community, pricing and other - notes for the major platforms as well as IBM OpenWhisk and - the [Fission.io](https://fission.io/) project. - -* [Microsoft Azure Functions vs. Google Cloud Functions vs. AWS Lambda](https://cloudacademy.com/blog/microsoft-azure-functions-vs-google-cloud-functions-fight-for-serverless-cloud-domination-continues/) - presents an overview of Azure Functions and how they compare to - Google Cloud Functions and AWS Lambda. +process for which platform to learn. +[Microsoft Azure Functions vs. Google Cloud Functions vs. AWS Lambda](https://cloudacademy.com/blog/microsoft-azure-functions-vs-google-cloud-functions-fight-for-serverless-cloud-domination-continues/) +presents an overview of Azure Functions and how they compare to +Google Cloud Functions and AWS Lambda. ### Serverless vendor lock-in? There is some concern by organizations and developers about vendor lock-in on serverless platforms. It is unclear if portability is worse for serverless than other infrastructure-as-a-service pieces, but still worth -thinking about ahead of time. These resources provide additional -perspectives on lock-in and using multiple cloud providers. - -* [On Serverless, Multi-Cloud, and Vendor Lock In](https://blog.symphonia.io/on-serverless-multi-cloud-and-vendor-lock-in-da930b3993f) - is an opinion piece that for *most* cases the additional work of - going multi-cloud is not worth the tradeoffs, therefore at this time - it's better to go for a single vendor such as AWS or Azure and optimize - on that platform. - -* [Why vendor lock-in with serverless isn’t what you think it is](https://medium.com/@PaulDJohnston/why-vendor-lock-in-with-serverless-isnt-what-you-think-it-is-d6be40fa9ca9) - recommends using a single vendor for now and stop worrying about - hedging your bets because it typically makes your infrastructure - significantly more complex. +thinking about ahead of time. +[Why vendor lock-in with serverless isn’t what you think it is](https://medium.com/@PaulDJohnston/why-vendor-lock-in-with-serverless-isnt-what-you-think-it-is-d6be40fa9ca9) +is a piece on this topic that recommends using a single vendor for +now and for organizations to stop worrying about hedging their bets +because it typically makes infrastructure significantly more complex. diff --git a/content/pages/05-deployment/39-aws-lambda.markdown b/content/pages/05-deployment/39-aws-lambda.markdown index fec7e2944..80efa5ef7 100644 --- a/content/pages/05-deployment/39-aws-lambda.markdown +++ b/content/pages/05-deployment/39-aws-lambda.markdown @@ -38,8 +38,7 @@ has support for both Python 2.7, 3.6 and 3.7. * [Zappa](https://github.com/Miserlou/Zappa) is a serverless framework for deploying Python web applications. It's a really slick project and used even by internal AWS developers for their own application - deployments. Be sure to [read the Zappa blog](https://blog.zappa.io/) - as well for walkthroughs and new feature announcements. + deployments. * [How to Setup a Serverless URL Shortener With API Gateway Lambda and DynamoDB on AWS](https://blog.ruanbekker.com/blog/2018/11/30/how-to-setup-a-serverless-url-shortener-with-api-gateway-lambda-and-dynamodb-on-aws/) builds a non-trivial URL shortener application as an example Python @@ -53,10 +52,6 @@ has support for both Python 2.7, 3.6 and 3.7. provides a screen capture of one developer deploying their application to Lambda. -* [Automated SQL Injection Testing of Serverless Functions On a Shoestring Budget (and Some Good Music)](https://www.puresec.io/blog/automated-sql-injection-testing-of-serverless-functions-on-a-shoestring-budget-and-some-good-music) - is an awesome operational security post that uses Python to test - for SQL injection vulnerabilities in serverless functions on AWS Lambda. - * [Building Scikit-Learn For AWS Lambda](https://serverlesscode.com/post/scikitlearn-with-amazon-linux-container/) follows up on the [Using Scikit-Learn In AWS Lambda](https://serverlesscode.com/post/deploy-scikitlearn-on-lamba/) @@ -70,14 +65,6 @@ has support for both Python 2.7, 3.6 and 3.7. * [Code Evaluation With AWS Lambda and API Gateway](https://realpython.com/blog/python/code-evaluation-with-aws-lambda-and-api-gateway/) shows how to develop a code evaluation API, to execute arbitrary code, with AWS Lambda and API Gateway. -* [Crawling thousands of products using AWS Lambda](https://engineering.21buttons.com/crawling-thousands-of-products-using-aws-lambda-80332e259de1) - gives a real-world example of where using Python, Selenium and - [headless Chrome](https://developers.google.com/web/updates/2017/04/headless-chrome) - on AWS Lambda could crawl thousands of pages to collect data - with each crawler running within its own Lambda Function. - -* [Going Serverless with AWS Lambda and API Gateway](http://blog.ryankelly.us/2016/08/07/going-serverless-with-aws-lambda-and-api-gateway.html) - ### General AWS Lambda resources * [Getting started with serverless on AWS](https://emshea.com/post/serverless-getting-started) @@ -85,11 +72,6 @@ has support for both Python 2.7, 3.6 and 3.7. guide created by a developer who used all of these bits to learn AWS services herself. -* [AWS Lambda Serverless Reference Architectures](http://www.allthingsdistributed.com/2016/06/aws-lambda-serverless-reference-architectures.html) - provides blueprints with diagrams of common architecture patterns that - developers use for their mobile backend, file processing, stream - processing and web application projects. - * [Security Overview of AWS Lambda](https://d1.awsstatic.com/whitepapers/Overview-AWS-Lambda-Security.pdf) (PDF file) covers their "Shared Responsibility Model" for security and compliance. Although the paper bills itself as an in-depth look at diff --git a/content/pages/05-deployment/40-azure-functions.markdown b/content/pages/05-deployment/40-azure-functions.markdown index 4cb8243e1..7f718f64a 100644 --- a/content/pages/05-deployment/40-azure-functions.markdown +++ b/content/pages/05-deployment/40-azure-functions.markdown @@ -44,6 +44,3 @@ in other Azure services. what is confusing to newcomers that hopefully will be addressed as Microsoft continues to work on their Azure platform. -* [Azure in Plain English](https://www.expeditedssl.com/azure-in-plain-english) - covers all of the Azure services and explains them because their - default names are often too vague to understand their purpose. diff --git a/content/pages/06-devops/00-devops.markdown b/content/pages/06-devops/00-devops.markdown index c1a95152a..f6053c2e4 100644 --- a/content/pages/06-devops/00-devops.markdown +++ b/content/pages/06-devops/00-devops.markdown @@ -35,8 +35,8 @@ tools and services for DevOps environments. which when used properly can enable continuous software delivery. * For an Atlassian-centric perspective on tooling, take a look at - this post on how to - [choose the right DevOps tools](http://blogs.atlassian.com/2016/03/how-to-choose-devops-tools/) + this guide on how to + [choose the right DevOps tools](https://www.atlassian.com/devops/devops-tools) which is biased towards their tools but still has some good insight such as using automated testing to provide immediate awareness of defects that require fixing. @@ -47,10 +47,10 @@ The following resources give advice and approaches for building the right teams, culture, processes and tools into software development organizations. * [DevOps vs. Platform Engineering](https://alexgaynor.net/2015/mar/06/devops-vs-platform-engineering/) - considers DevOps an ad hoc approach to developing software while building - a platform is a strict contract. I see this as "DevOps is a process", - while a "platform is code". Running code is better than any organizational - process. + considers DevOps to be an ad hoc approach to developing software while + building a platform is a strict contract. I see this as "DevOps is a + process", while a "platform is code". Running code is better than any + organizational process. * The open source [PagerDuty Incident Response guide](https://response.pagerduty.com/) is the @@ -58,7 +58,7 @@ teams, culture, processes and tools into software development organizations. their services running and putting them out for other developers to consume. Highly recommended. -* [Introduction to DevOps and Software Delivery Performance](https://www.stridenyc.com/blog/introduction-to-devops-and-software-delivery-performance) +* [Introduction to DevOps and Software Delivery Performance](https://www.stridenyc.com/blog/devops-and-software-delivery-performance) explains the four key delivery metrics of Delivery Lead Time, Deployment Frequency, Time to Restore Service, and Change Fail Rate, and then gives a high-level overview of technical, process and @@ -88,10 +88,6 @@ teams, culture, processes and tools into software development organizations. your pager goes off, ownership and how startups can be different from large companies with their incident responses. -* [Bing: Continuous Delivery](http://stories.visualstudio.com/bing-continuous-delivery/) - is an impressive visual story that explains the practices for how their - team delivers updates to the search engine. - * [Why are we racing to DevOps?](http://www.cio.com/article/3015237/application-development/why-are-we-racing-to-devops.html) is a very high level summary of the benefits of DevOps to IT organizations. It's not specific to Python and doesn't dive into the details, but it's diff --git a/content/pages/06-devops/01-monitoring.markdown b/content/pages/06-devops/01-monitoring.markdown index 95efe2cf4..50021eecf 100644 --- a/content/pages/06-devops/01-monitoring.markdown +++ b/content/pages/06-devops/01-monitoring.markdown @@ -112,7 +112,7 @@ Application Performance Monitoring (APM) * [New Relic](http://newrelic.com/) provides application and database monitoring as well as plug ins for capturing and analyzing data about - other devleoper tools in your stack, such as [Twilio](/twilio.html). + other developer tools in your stack, such as [Twilio](/twilio.html). * [Opbeat](https://opbeat.com) Built for django. Opbeat combines performance metrics, release tracking, and error logging into a single simple service. * [Scout](https://scoutapp.com/python-monitoring) monitors the performance of Django and Flask apps, auto-instrumenting views, SQL queries, templates, and more. @@ -146,11 +146,6 @@ Incident Management * [The Virtues of Monitoring](http://www.paperplanes.de/2011/1/5/the_virtues_of_monitoring.html) -* [Effortless Monitoring with collectd, Graphite, and Docker](http://blog.docker.io/2013/07/effortless-monitoring-with-collectd-graphite-and-docker/) - -* [Practical Guide to StatsD/Graphite Monitoring](http://matt.aimonetti.net/posts/2013/06/26/practical-guide-to-graphite-monitoring/) - is a detailed guide with code examples for monitoring infrastructure. - * Bit.ly describes the "[10 Things They Forgot to Monitor](http://word.bitly.com/post/74839060954/ten-things-to-monitor)" beyond the standard metrics such as disk & memory usage. diff --git a/content/pages/06-devops/06-web-app-performance.markdown b/content/pages/06-devops/06-web-app-performance.markdown index faa195727..8a954546f 100644 --- a/content/pages/06-devops/06-web-app-performance.markdown +++ b/content/pages/06-devops/06-web-app-performance.markdown @@ -49,10 +49,6 @@ database queries, page size and many other factors. is a 20 minute code-first demo that shows how to get a realistic estimate for how many requests per second your web application will be able to handle. -* [How to Interpret Site Performance Tests](https://fly.io/articles/how-to-understand-performance-tests/) - covers the difference between client, page and connection speed tests - as well as a bit on caching performance. - * [Practical scaling techniques for websites](https://hackernoon.com/practical-scaling-techniques-for-web-sites-554a38dbd492) examines how to improve your website performance with asynchronous [task queues](/task-queues.html), [database](/databases.html) optimization diff --git a/content/pages/06-devops/11-caching.markdown b/content/pages/06-devops/11-caching.markdown index 5a8e94abf..c8aebbbf1 100644 --- a/content/pages/06-devops/11-caching.markdown +++ b/content/pages/06-devops/11-caching.markdown @@ -45,19 +45,10 @@ A cache can be created for multiple layers of the stack. reading even though the author is describing his Microsoft code as the impetus for writing the content. -* While caching is a useful technique in many situations, it's important - to also note that there are - [downsides to caching](https://msol.io/blog/tech/2015/09/05/youre-probably-wrong-about-caching/) - that many developers fail to take into consideration. - * [Caching at Reddit](https://redditblog.com/2017/1/17/caching-at-reddit/) covers monitoring, tuning and scaling for the very high scale [Reddit.com](https://www.reddit.com/) website. -* [Mastering HTTP caching](https://blog.fortrabbit.com/mastering-http-caching) - provides more advanced advice on caching dynamic as well as static - content via CDNs and other configurations. - ## Caching learning checklist diff --git a/content/pages/06-devops/14-logging.markdown b/content/pages/06-devops/14-logging.markdown index 3fa5b0e30..9738d98fc 100644 --- a/content/pages/06-devops/14-logging.markdown +++ b/content/pages/06-devops/14-logging.markdown @@ -55,8 +55,9 @@ certain threshold. There are libraries for most major languages, including python. Saves data in Elasticache. -* [Logstash](http://logstash.net/) Similar to Graylog2, logstash offers - features to programmatically configure log data workflows. +* [Logstash](https://www.elastic.co/guide/en/logstash/current/index.html). + Similar to Graylog2, logstash offers features to programmatically + configure log data workflows. * [Scribe](https://github.com/facebook/scribe) A project written by Facebook to aggregate logs. It's designed to run on multiple servers and scale with @@ -112,7 +113,7 @@ certain threshold. * [Good logging practice in Python](http://victorlin.me/posts/2012/08/26/good-logging-practice-in-python) shows how to use the standard library to log data from your application. Definitely worth a read as most applications do not log nearly enough - output to help debuggin when things go wrong, or to determine if something + output to help debugging when things go wrong, or to determine if something is going wrong. * [Structured Logging: The Best Friend You’ll Want When Things Go Wrong](https://engineering.grab.com/structured-logging) diff --git a/content/pages/06-devops/18-web-analytics.markdown b/content/pages/06-devops/18-web-analytics.markdown index 28c0a58a9..6430b1057 100644 --- a/content/pages/06-devops/18-web-analytics.markdown +++ b/content/pages/06-devops/18-web-analytics.markdown @@ -38,9 +38,10 @@ application before taking some action, such as purchasing your service. ## Open source web analytics projects -* [Piwik](http://piwik.org/) is a web analytics platform you can host yourself. - Piwik is a solid choice if you cannot use Google Analytics or want to - customize your own web analytics platform. +* [Matoma](https://matomo.org/) (formerly Piwik), is a web analytics + platform that you can host yourself. Matoma is a solid choice if you + cannot use Google Analytics or want to customize your own web analytics + software. * [Shynet](https://github.com/milesmcc/shynet) is a lightweight, privacy-friendly cookie-free web analytics application written in Python. @@ -109,10 +110,6 @@ application before taking some action, such as purchasing your service. is not for everyone but it is really useful if you want to avoid the Google data trap. -* This beginner's guide to - [math and stats behind web analytics](http://www.seotakeaways.com/beginners-guide-maths-stats-web-analytics/) - provides some context for understanding and reasoning about web traffic. - * [An Analytics Primer for Developers](https://hacks.mozilla.org/2015/03/an-analytics-primer-for-developers/) by Mozilla explains what to track, choosing an analytics platform and how to serve up the analytics JavaScript asynchronously. @@ -142,8 +139,8 @@ application before taking some action, such as purchasing your service. ## Web analytics learning checklist -1. Add Google Analytics or Piwik to your application. Both are free and while - Piwik is not as powerful as Google Analytics you can self-host the +1. Add Google Analytics or Matoma to your application. Both are free and while + Matoma is not as powerful as Google Analytics you can self-host the application which is the only option in many environments. 1. Think critically about the factors that will make your application diff --git a/content/pages/10-working/18-developer-demos.md b/content/pages/10-working/18-developer-demos.md new file mode 100644 index 000000000..a195edfdc --- /dev/null +++ b/content/pages/10-working/18-developer-demos.md @@ -0,0 +1,29 @@ +title: Demoing to Software Developers +category: page +slug: demo-software-developers +sortorder: 1018 +toc: False +sidebartitle: Developer Demos +meta: How to give technical demos to software developers. + + +Creating and executing an appealing technical demo to an audience of software +developers is a ton of work, but there is no better way to get people +legitimately interested in your product if you land a great demo with +an appropriate audience. The inherent difficulty involved in exceptional +technical demos also creates a large barrier that prevent others from simply +copying your work, which can happen with technical blog content. + +To achieve a standout technical demo you must: + +1. show how to easily solve a difficult technical problem +1. speak plainly but accurately +1. do it live, including writing code if required +1. tell a story with a narrative arc +1. rehearse constantly, both the happy path and recovering from errors + + +Examples: +* [Twilio Phone Calls Demo at NY Tech Meetup (2010)](https://www.youtube.com/watch?v=-VuXIgp9S7o) +* [Concurrency from the Ground Up (2015)](https://www.youtube.com/watch?v=MCs5OvhV9S4) +* [The Mother of All Demos (1968)](https://www.youtube.com/watch?v=yJDv-zdhzMY) diff --git a/content/pages/examples/django/django-template-context.markdown b/content/pages/examples/django/django-template-context.markdown index 5600af98b..9bfe9e562 100644 --- a/content/pages/examples/django/django-template-context.markdown +++ b/content/pages/examples/django/django-template-context.markdown @@ -582,7 +582,6 @@ __author__ = "Bas van Oostveen ", __contributors__ = [ "Antonio Cavedoni " "Stefano J. Attardi ", - "limodou ", "Carlo C8E Miron", "Andre Campos ", "Justin Findlay ", diff --git a/content/pages/examples/django/django-template-loader.markdown b/content/pages/examples/django/django-template-loader.markdown index 6eb9853b6..9098f4881 100644 --- a/content/pages/examples/django/django-template-loader.markdown +++ b/content/pages/examples/django/django-template-loader.markdown @@ -178,7 +178,6 @@ __author__ = "Bas van Oostveen ", __contributors__ = [ "Antonio Cavedoni " "Stefano J. Attardi ", - "limodou ", "Carlo C8E Miron", "Andre Campos ", "Justin Findlay ", diff --git a/content/pages/examples/flask/flask-app-badrequest.markdown b/content/pages/examples/flask/flask-app-badrequest.markdown index 76ff30b02..a2c3c3abd 100644 --- a/content/pages/examples/flask/flask-app-badrequest.markdown +++ b/content/pages/examples/flask/flask-app-badrequest.markdown @@ -20,7 +20,7 @@ that accepts POSTs. and ImmutableDict are several other callables with code examples from the same `flask.app` package. -You should read up on these subjects along with these `BadRequest` examples: +These subjects go along with the `BadRequest` code examples: * [web development](/web-development.html) and [web design](/web-design.html) * [Flask](/flask.html) and [web framework](/web-frameworks.html) concepts @@ -90,7 +90,6 @@ from ..const import ( ## ... source file abbreviated to get to BadRequest examples ... - API_SELECT_COLUMNS_RIS_KEY, API_SHOW_COLUMNS_RES_KEY, API_SHOW_COLUMNS_RIS_KEY, API_SHOW_TITLE_RES_KEY, @@ -99,6 +98,7 @@ from ..const import ( PERMISSION_PREFIX, ) from ..exceptions import FABException, InvalidOrderByColumnFABException +from ..hooks import get_before_request_hooks, wrap_route_handler_with_hooks from ..security.decorators import permission_name, protect log = logging.getLogger(__name__) @@ -149,7 +149,7 @@ def rison(schema=None): ## Example 2 from Flask-WTF [Flask-WTF](https://github.com/lepture/flask-wtf) -([project documentation](https://flask-wtf.readthedocs.io/en/stable/) +([project documentation](https://flask-wtf.readthedocs.io/) and [PyPI page](https://pypi.org/project/Flask-WTF/)) provides a bridge between [Flask](/flask.html) and the the @@ -274,11 +274,11 @@ def get_error_description(exception): except AttributeError: return str(exception) if isinstance(exception, Forbidden) and description == Forbidden.description: - return _("You are not allowed to access this page.") + return _('You are not allowed to access this page.') elif isinstance(exception, NotFound) and description == NotFound.description: return _("The page you are looking for doesn't exist.") ~~ elif isinstance(exception, BadRequest) and description == BadRequest.description: - return _("The request was invalid or contained invalid arguments.") + return _('The request was invalid or contained invalid arguments.') else: return str(description) diff --git a/content/pages/examples/flask/flask-app-flask.markdown b/content/pages/examples/flask/flask-app-flask.markdown index 33f54b1ef..0702a72ad 100644 --- a/content/pages/examples/flask/flask-app-flask.markdown +++ b/content/pages/examples/flask/flask-app-flask.markdown @@ -19,7 +19,7 @@ application's functionality, including URL rounting, and ImmutableDict are several other callables with code examples from the same `flask.app` package. -These subjects go along with the `Flask` code examples: +You should read up on these subjects along with these `Flask` examples: * [web development](/web-development.html) and [web design](/web-design.html) * [Flask](/flask.html) and [web framework](/web-frameworks.html) concepts @@ -167,23 +167,23 @@ def test_that_request_path_hijacking_works_properly(): def test_theme_fallback_config(): - app = create_ctfd() + + class ThemeFallbackConfig(TestingConfig): + THEME_FALLBACK = False + + app = create_ctfd(config=ThemeFallbackConfig) try: - os.mkdir(os.path.join(app.root_path, "themes", "foo")) + os.mkdir(os.path.join(app.root_path, "themes", "foo_fallback")) except OSError: pass with app.app_context(): - set_config("ctf_theme", "foo") + app.config["THEME_FALLBACK"] = False + set_config("ctf_theme", "foo_fallback") assert app.config["THEME_FALLBACK"] == False with app.test_client() as client: try: r = client.get("/") - except TemplateNotFound: - pass - try: - r = client.get("/themes/foo/static/js/pages/main.dev.js") - except TemplateNotFound: ## ... source file continues with no further Flask examples... @@ -204,66 +204,159 @@ forms, and internationalization support. Flask App Builder is provided under the [BSD 3-Clause "New" or "Revised" license](https://github.com/dpgaspar/Flask-AppBuilder/blob/master/LICENSE). -[**Flask AppBuilder / flask_appbuilder / tests / test_mongoengine.py**](https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/tests/test_mongoengine.py) +[**Flask AppBuilder / flask_appbuilder / tests / test_fab_cli.py**](https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/tests/test_fab_cli.py) ```python -# test_mongoengine.py -from flask_appbuilder.views import CompactCRUDMixin, MasterDetailView -from flask_mongoengine import MongoEngine -import jinja2 -from nose.tools import eq_, ok_ +# test_fab_cli.py +import glob +import json +import logging +import os +import tempfile + +from click.testing import CliRunner +~~from flask import Flask +from flask_appbuilder import AppBuilder, SQLA +from flask_appbuilder.cli import ( + create_app, + create_permissions, + create_user, + export_roles, + import_roles, + list_users, + list_views, + reset_password, +) from .base import FABTestCase -from .mongoengine.models import Model1, Model2 logging.basicConfig(format="%(asctime)s:%(levelname)s:%(name)s:%(message)s") logging.getLogger().setLevel(logging.DEBUG) +log = logging.getLogger(__name__) +APP_DIR = "myapp" -DEFAULT_INDEX_STRING = "Welcome" -INVALID_LOGIN_STRING = "Invalid login" -ACCESS_IS_DENIED = "Access is Denied" -UNIQUE_VALIDATION_STRING = "Already exists" -NOTNULL_VALIDATION_STRING = "This field is required" -DEFAULT_ADMIN_USER = "admin" -DEFAULT_ADMIN_PASSWORD = "general" -log = logging.getLogger(__name__) +class FlaskTestCase(FABTestCase): + def setUp(self): + pass -class FlaskTestCase(FABTestCase): +## ... source file abbreviated to get to Flask examples ... + + + self.assertIn("User bob created.", result.output) + + result = runner.invoke(list_users, []) + self.assertIn("bob", result.output) + + runner.invoke(create_permissions, []) + + runner.invoke(reset_password, ["--username=bob", "--password=bar"]) + + def test_list_views(self): + os.environ["FLASK_APP"] = "app:app" + runner = CliRunner() + with runner.isolated_filesystem(): + result = runner.invoke(list_views, []) + self.assertIn("List of registered views", result.output) + self.assertIn(" Route:/api/v1/security", result.output) + + +class SQLAlchemyImportExportTestCase(FABTestCase): def setUp(self): -~~ from flask import Flask - from flask_appbuilder import AppBuilder - from flask_appbuilder.models.mongoengine.interface import MongoEngineInterface - from flask_appbuilder import ModelView - from flask_appbuilder.security.mongoengine.manager import SecurityManager - -~~ self.app = Flask(__name__) - self.app.jinja_env.undefined = jinja2.StrictUndefined - self.basedir = os.path.abspath(os.path.dirname(__file__)) - self.app.config["MONGODB_SETTINGS"] = {"DB": "test"} - self.app.config["CSRF_ENABLED"] = False - self.app.config["SECRET_KEY"] = "thisismyscretkey" - self.app.config["WTF_CSRF_ENABLED"] = False - - self.db = MongoEngine(self.app) - self.appbuilder = AppBuilder(self.app, security_manager_class=SecurityManager) - - class Model2View(ModelView): - datamodel = MongoEngineInterface(Model2) - list_columns = [ - "field_integer", - "field_float", - "field_string", - "field_method", - "group.field_string", - ] - edit_form_query_rel_fields = { - "group": [["field_string", FilterEqual, "G2"]] - } - add_form_query_rel_fields = {"group": [["field_string", FilterEqual, "G1"]]} - add_exclude_columns = ["excluded_string"] + with open("flask_appbuilder/tests/data/roles.json", "r") as fd: + self.expected_roles = json.loads(fd.read()) + + def test_export_roles(self): + with tempfile.TemporaryDirectory() as tmp_dir: +~~ app = Flask("src_app") + app.config.from_object("flask_appbuilder.tests.config_security") + app.config[ + "SQLALCHEMY_DATABASE_URI" + ] = f"sqlite:///{os.path.join(tmp_dir, 'src.db')}" + db = SQLA(app) + app_builder = AppBuilder(app, db.session) # noqa: F841 + cli_runner = app.test_cli_runner() + + path = os.path.join(tmp_dir, "roles.json") + + export_result = cli_runner.invoke(export_roles, [f"--path={path}"]) + + self.assertEqual(export_result.exit_code, 0) + self.assertTrue(os.path.exists(path)) + + with open(path, "r") as fd: + resulting_roles = json.loads(fd.read()) + + for expected_role in self.expected_roles: + match = [ + r for r in resulting_roles if r["name"] == expected_role["name"] + ] + self.assertTrue(match) + resulting_role = match[0] + resulting_role_permission_view_menus = { + (pvm["permission"]["name"], pvm["view_menu"]["name"]) + for pvm in resulting_role["permissions"] + } + expected_role_permission_view_menus = { + (pvm["permission"]["name"], pvm["view_menu"]["name"]) + for pvm in expected_role["permissions"] + } + self.assertEqual( + resulting_role_permission_view_menus, + expected_role_permission_view_menus, + ) + + def test_export_roles_filename(self): + with tempfile.TemporaryDirectory() as tmp_dir: +~~ app = Flask("src_app") + app.config.from_object("flask_appbuilder.tests.config_security") + + app.config[ + "SQLALCHEMY_DATABASE_URI" + ] = f"sqlite:///{os.path.join(tmp_dir, 'src.db')}" + db = SQLA(app) + app_builder = AppBuilder(app, db.session) # noqa: F841 + + owd = os.getcwd() + os.chdir(tmp_dir) + cli_runner = app.test_cli_runner() + export_result = cli_runner.invoke(export_roles) + os.chdir(owd) + + self.assertEqual(export_result.exit_code, 0) + self.assertGreater( + len(glob.glob(os.path.join(tmp_dir, "roles_export_*"))), 0 + ) + + def test_import_roles(self): + with tempfile.TemporaryDirectory() as tmp_dir: +~~ app = Flask("dst_app") + app.config[ + "SQLALCHEMY_DATABASE_URI" + ] = f"sqlite:///{os.path.join(tmp_dir, 'dst.db')}" + db = SQLA(app) + app_builder = AppBuilder(app, db.session) + cli_runner = app.test_cli_runner() + + path = os.path.join(tmp_dir, "roles.json") + + with open(path, "w") as fd: + fd.write(json.dumps(self.expected_roles)) + + self.assertEqual(len(app_builder.sm.get_all_roles()), 2) + + import_result = cli_runner.invoke(import_roles, [f"--path={path}"]) + self.assertEqual(import_result.exit_code, 0) + + resulting_roles = app_builder.sm.get_all_roles() + + for expected_role in self.expected_roles: + match = [r for r in resulting_roles if r.name == expected_role["name"]] + self.assertTrue(match) + resulting_role = match[0] + ## ... source file continues with no further Flask examples... @@ -308,7 +401,7 @@ from flaskbb.plugins.models import PluginRegistry from flaskbb.plugins.utils import remove_zombie_plugins_from_db, template_hook from flaskbb.user.models import Guest, User from flaskbb.utils.helpers import (app_config_from_env, crop_title, - format_date, format_datetime, + format_date, format_time, format_datetime, forum_is_unread, get_alembic_locations, get_flaskbb_config, is_online, mark_online, render_template, time_since, time_utcnow, @@ -380,7 +473,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) backend, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling and many others. flask-base is provided as open source under the @@ -677,7 +770,7 @@ class HTTPAuthTestCase(unittest.TestCase): is an example application that ties together the [intTellInput.js](https://github.com/jackocnr/intl-tel-input) JavaScript plugin with the -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) form-handling +[Flask-WTF](https://flask-wtf.readthedocs.io/) form-handling library. flask-phone-input is provided as open source under the [MIT license](https://github.com/miguelgrinberg/flask-phone-input/blob/1a1c227c044474ce0fe133493d7f8b0fb8312409/LICENSE). @@ -729,7 +822,7 @@ def index(): starter project to build a software-as-a-service (SaaS) web application in [Flask](/flask.html), with [Stripe](/stripe.html) for billing. The boilerplate relies on many common Flask extensions such as -[Flask-WTF](https://flask-wtf.readthedocs.io/en/latest/), +[Flask-WTF](https://flask-wtf.readthedocs.io/), [Flask-Login](https://flask-login.readthedocs.io/en/latest/), [Flask-Admin](https://flask-admin.readthedocs.io/en/latest/), and many others. The project is provided as open source under the @@ -773,7 +866,83 @@ from app.models import User ``` -## Example 12 from Flask-SocketIO +## Example 12 from Flask-Security-Too +[Flask-Security-Too](https://github.com/Flask-Middleware/flask-security/) +([PyPi page](https://pypi.org/project/Flask-Security-Too/) and +[project documentation](https://flask-security-too.readthedocs.io/en/stable/)) +is a maintained fork of the original +[Flask-Security](https://github.com/mattupstate/flask-security) project that +makes it easier to add common security features to [Flask](/flask.html) +web applications. A few of the critical goals of the Flask-Security-Too +project are ensuring JavaScript client-based single-page applications (SPAs) +can work securely with Flask-based backends and that guidance by the +[OWASP](https://owasp.org/) organization is followed by default. + +The Flask-Security-Too project is provided as open source under the +[MIT license](https://github.com/Flask-Middleware/flask-security/blob/master/LICENSE). + +[**Flask-Security-Too / flask_security / utils.py**](https://github.com/Flask-Middleware/flask-security/blob/master/flask_security/./utils.py) + +```python +# utils.py + flash, + g, + request, + render_template, + session, + url_for, +) +from flask.json import JSONEncoder +from flask_login import login_user as _login_user +from flask_login import logout_user as _logout_user +from flask_login import current_user +from flask_login import COOKIE_NAME as REMEMBER_COOKIE_NAME +from flask_principal import AnonymousIdentity, Identity, identity_changed, Need +from flask_wtf import csrf +from wtforms import ValidationError +from itsdangerous import BadSignature, SignatureExpired +from werkzeug import __version__ as werkzeug_version +from werkzeug.local import LocalProxy +from werkzeug.datastructures import MultiDict + +from .quart_compat import best, get_quart_status +from .proxies import _security, _datastore, _pwd_context, _hashing_context +from .signals import user_authenticated + +if t.TYPE_CHECKING: # pragma: no cover +~~ from flask import Flask, Response + from .datastore import User + +SB = t.Union[str, bytes] + + +localize_callback = LocalProxy(lambda: _security.i18n_domain.gettext) + +FsPermNeed = partial(Need, "fsperm") +FsPermNeed.__doc__ = """A need with the method preset to `"fsperm"`.""" + + +def _(translate): + return translate + + +def get_request_attr(name: str) -> t.Any: + return getattr(_request_ctx_stack.top, name, None) + + +def set_request_attr(name, value): + return setattr(_request_ctx_stack.top, name, value) + + +if get_quart_status(): # pragma: no cover + + +## ... source file continues with no further Flask examples... + +``` + + +## Example 13 from Flask-SocketIO [Flask-SocketIO](https://github.com/miguelgrinberg/Flask-SocketIO) ([PyPI package information](https://pypi.org/project/Flask-SocketIO/), [official tutorial](https://blog.miguelgrinberg.com/post/easy-websockets-with-flask-and-gevent) @@ -889,7 +1058,7 @@ def connect(): ``` -## Example 13 from Flask-User +## Example 14 from Flask-User [Flask-User](https://github.com/lingthio/Flask-User) ([PyPI information](https://pypi.org/project/Flask-User/) and @@ -972,7 +1141,7 @@ class UserManager(UserManager__Settings, UserManager__Utils, UserManager__Views) ``` -## Example 14 from Flask-VueJs-Template +## Example 15 from Flask-VueJs-Template [Flask-VueJs-Template](https://github.com/gtalarico/flask-vuejs-template) ([demo site](https://flask-vuejs-template.herokuapp.com/)) is a minimal [Flask](/flask.html) boilerplate starter project that @@ -1013,7 +1182,7 @@ def index_client(): ``` -## Example 15 from Flasky +## Example 16 from Flasky [Flasky](https://github.com/miguelgrinberg/flasky) is a wonderful example application by [Miguel Grinberg](https://github.com/miguelgrinberg) that he builds @@ -1077,7 +1246,7 @@ def create_app(config_name): ``` -## Example 16 from Datadog Flask Example App +## Example 17 from Datadog Flask Example App The [Datadog Flask example app](https://github.com/DataDog/trace-examples/tree/master/python/flask) contains many examples of the [Flask](/flask.html) core functions available to a developer using the [web framework](/web-frameworks.html). @@ -1138,7 +1307,7 @@ def before_request(): ``` -## Example 17 from keras-flask-deploy-webapp +## Example 18 from keras-flask-deploy-webapp The [keras-flask-deploy-webapp](https://github.com/mtobeiyf/keras-flask-deploy-webapp) project combines the [Flask](/flask.html) [web framework](/web-frameworks.html) @@ -1202,7 +1371,7 @@ def model_predict(img, model): ``` -## Example 18 from sandman2 +## Example 19 from sandman2 [sandman2](https://github.com/jeffknupp/sandman2) ([project documentation](https://sandman2.readthedocs.io/en/latest/) and @@ -1283,7 +1452,7 @@ def get_app( ``` -## Example 19 from Science Flask +## Example 20 from Science Flask [Science Flask](https://github.com/danielhomola/science_flask) is a [Flask](/flask.html)-powered web application for online scientific research tools. The project was built as a template @@ -1352,14 +1521,14 @@ def create_celery_app(): ``` -## Example 20 from ShortMe +## Example 21 from ShortMe [ShortMe](https://github.com/AcrobaticPanicc/ShortMe-URL-Shortener) is a [Flask](/flask.html) app that creates a shortened URL that redirects to another, typically much longer, URL. The project is provided as open source under the [MIT license](https://github.com/AcrobaticPanicc/ShortMe-URL-Shortener/blob/main/LICENSE). -[**ShortMe / app / setup.py**](https://github.com/AcrobaticPanicc/ShortMe-URL-Shortener/blob/main/app/./setup.py) +[**ShortMe / app / setup / setup.py**](https://github.com/AcrobaticPanicc/ShortMe-URL-Shortener/blob/main/app/setup/setup.py) ```python # setup.py @@ -1369,23 +1538,23 @@ import os from flask_restful import Api from dotenv import load_dotenv -from app.db.extensions import db -from app.db.models import AuthToken -from app.views.index.index import index_blueprint -from app.views.internal.redirect_to_url import redirect_to_url_blueprint -from app.views.internal.favicon import app_blueprint -from app.views.internal.send_verification_code import send_otp_blueprint -from app.views.internal.shorten_url import shorten_url_blueprint -from app.views.your_short_url.your_short_url import your_short_url_blueprint -from app.views.total_clicks.total_clicks import total_clicks_blueprint -from app.views.error.error import error_blueprint -from app.views.page_not_found.page_not_found import page_not_found_blueprint -from app.views.api_doc.api_doc import api_doc_blueprint -from app.views.get_token.get_token import get_token_blueprint -from app.views.your_api_token.your_api_token import your_api_token_blueprint -from app.views.verify_code.verify_code import verify_code_blueprint - -from app.api.api import Shorten, TotalClicks, GetToken +from app.server.db.extensions import db +from app.server.db.models import AuthToken +from app.server.routes.index import index_blueprint +from app.server.routes.internal.redirect_to_url import redirect_to_url_blueprint +from app.server.routes.internal.favicon import app_blueprint +from app.server.routes.internal.send_verification_code import send_otp_blueprint +from app.server.routes.internal.shorten_url import shorten_url_blueprint +from app.server.routes.your_short_url import your_short_url_blueprint +from app.server.routes.total_clicks import total_clicks_blueprint +from app.server.routes.error import error_blueprint +from app.server.routes.page_not_found import page_not_found_blueprint +from app.server.routes.api_doc import api_doc_blueprint +from app.server.routes.get_token import get_token_blueprint +from app.server.routes.your_api_token import your_api_token_blueprint +from app.server.routes.verify_code import verify_code_blueprint + +from app.server.api.api import Shorten, TotalClicks, GetToken def create_app(config_file): @@ -1393,7 +1562,7 @@ def create_app(config_file): project_folder = os.path.expanduser(app_path) load_dotenv(os.path.join(project_folder, '.env')) -~~ app = Flask(__name__) +~~ app = Flask(__name__, template_folder='../client/templates', static_folder='../client/static') api = Api(app) app.config.from_pyfile(config_file) @@ -1425,7 +1594,7 @@ def create_app(config_file): ``` -## Example 21 from tedivms-flask +## Example 22 from tedivms-flask [tedivm's flask starter app](https://github.com/tedivm/tedivms-flask) is a base of [Flask](/flask.html) code and related projects such as [Celery](/celery.html) which provides a template to start your own @@ -1555,7 +1724,7 @@ def create_app(extra_config_settings={}): ``` -## Example 22 from trape +## Example 23 from trape [trape](https://github.com/jofpin/trape) is a research tool for tracking people's activities that are logged digitally. The tool uses [Flask](/flask.html) to create a web front end to view aggregated data @@ -1577,6 +1746,8 @@ from core.db import Database import os import sys import platform +import urllib +import requests from multiprocessing import Process trape = core.stats.trape @@ -1587,13 +1758,11 @@ db = Database() class victim_server(object): @app.route("/" + trape.victim_path) def homeVictim(): - opener = urllib2.build_opener() - headers = victim_headers(request.user_agent) - opener.addheaders = headers + r = requests.get(trape.url_to_clone, headers=victim_headers2(request.user_agent)) if (trape.type_lure == 'local'): html = assignScripts(victim_inject_code(render_template("/" + trape.url_to_clone), 'payload', '/', trape.gmaps, trape.ipinfo)) else: - html = assignScripts(victim_inject_code(opener.open(trape.url_to_clone).read(), 'payload', trape.url_to_clone, trape.gmaps, trape.ipinfo)) + html = assignScripts(victim_inject_code(r.content, 'payload', trape.url_to_clone, trape.gmaps, trape.ipinfo)) ## ... source file continues with no further Flask examples... diff --git a/content/pages/examples/flask/flask-app-headers.markdown b/content/pages/examples/flask/flask-app-headers.markdown index cbbeeabc3..e279fec3c 100644 --- a/content/pages/examples/flask/flask-app-headers.markdown +++ b/content/pages/examples/flask/flask-app-headers.markdown @@ -47,10 +47,12 @@ import random import string import uuid from collections import namedtuple +from contextlib import contextmanager from unittest.mock import Mock, patch import requests from flask.testing import FlaskClient +from freezegun import freeze_time from sqlalchemy.engine.url import make_url from sqlalchemy_utils import drop_database ~~from werkzeug.datastructures import Headers @@ -63,6 +65,7 @@ from CTFd.models import ( ChallengeComments, ChallengeFiles, Challenges, + ChallengeTopics, Comments, Fails, Fields, @@ -78,11 +81,14 @@ from CTFd.models import ( TeamComments, Teams, Tokens, + Topics, Tracking, Unlocks, UserComments, Users, ) +from CTFd.utils import set_config +from tests.constants.time import FreezeTimes text_type = str binary_type = bytes @@ -104,25 +110,25 @@ class CTFdTestClient(FlaskClient): return super(CTFdTestClient, self).open(*args, **kwargs) -def create_ctfd( - ctf_name="CTFd", - ctf_description="CTF description", - name="admin", - email="admin@examplectf.com", - password="password", - user_mode="users", - setup=True, - enable_plugins=False, - application_root="/", - config=TestingConfig, -): - if enable_plugins: - config.SAFE_MODE = False - else: - config.SAFE_MODE = True +class ctftime: + @contextmanager + def init(): + try: + set_config("start", FreezeTimes.START) + set_config("end", FreezeTimes.END) + yield + finally: + set_config("start", None) + set_config("end", None) - config.APPLICATION_ROOT = application_root - url = make_url(config.SQLALCHEMY_DATABASE_URI) + @contextmanager + def not_started(): + try: + freezer = freeze_time(FreezeTimes.NOT_STARTED) + frozen_time = freezer.start() + yield frozen_time + finally: + freezer.stop() ## ... source file continues with no further Headers examples... @@ -145,8 +151,6 @@ Flask RESTX is provided as open source under the ```python # api.py -from __future__ import unicode_literals - import difflib import inspect from itertools import chain @@ -163,7 +167,10 @@ from types import MethodType from flask import url_for, request, current_app from flask import make_response as original_flask_make_response -from flask.helpers import _endpoint_from_view_func +try: + from flask.helpers import _endpoint_from_view_func +except ImportError: + from flask.scaffold import _endpoint_from_view_func from flask.signals import got_request_exception from jsonschema import RefResolver @@ -177,7 +184,13 @@ from werkzeug.exceptions import ( NotAcceptable, InternalServerError, ) -from werkzeug.wrappers import BaseResponse + +from werkzeug import __version__ as werkzeug_version + +if werkzeug_version.split('.')[0] >= '2': + from werkzeug.wrappers import Response as BaseResponse +else: + from werkzeug.wrappers import BaseResponse from . import apidoc from .mask import ParseError, MaskError @@ -189,28 +202,22 @@ from .utils import default_id, camel_to_dash, unpack from .representations import output_json from ._http import HTTPStatus -RE_RULES = re.compile("(<.*>)") - -HEADERS_BLACKLIST = ("Content-Length",) - -DEFAULT_REPRESENTATIONS = [("application/json", output_json)] - ## ... source file abbreviated to get to Headers examples ... + if self._has_fr_route(): + try: return self.handle_error(e) except Exception as f: return original_handler(f) return original_handler(e) def handle_error(self, e): - got_request_exception.send(current_app._get_current_object(), exception=e) - if ( not isinstance(e, HTTPException) and current_app.propagate_exceptions - and not isinstance(e, tuple(self.error_handlers.keys())) + and not isinstance(e, tuple(self._own_and_child_error_handlers.keys())) ): exc_type, exc_value, tb = sys.exc_info() @@ -234,6 +241,8 @@ DEFAULT_REPRESENTATIONS = [("application/json", output_json)] ) break else: + got_request_exception.send(current_app._get_current_object(), exception=e) + if isinstance(e, HTTPException): code = HTTPStatus(e.code) if include_message_in_response: @@ -247,8 +256,6 @@ DEFAULT_REPRESENTATIONS = [("application/json", output_json)] else: code = HTTPStatus.INTERNAL_SERVER_ERROR if include_message_in_response: - default_data = { - "message": code.phrase, ## ... source file continues with no further Headers examples... diff --git a/content/pages/examples/flask/flask-app-immutabledict.markdown b/content/pages/examples/flask/flask-app-immutabledict.markdown index 899031de3..d2a334fa8 100644 --- a/content/pages/examples/flask/flask-app-immutabledict.markdown +++ b/content/pages/examples/flask/flask-app-immutabledict.markdown @@ -20,7 +20,7 @@ so that values cannot be modified after initially being set. and Headers are several other callables with code examples from the same `flask.app` package. -You should read up on these subjects along with these `ImmutableDict` examples: +These topics are also useful while reading the `ImmutableDict` examples: * [web development](/web-development.html) and [web design](/web-design.html) * [Flask](/flask.html) and [web framework](/web-frameworks.html) concepts @@ -89,7 +89,7 @@ DEFAULTS = { def load_config(only_defaults=False, override=None): - data = dict(DEFAULTS, **INTERNAL_DEFAULTS) + data = DEFAULTS | INTERNAL_DEFAULTS if not only_defaults: path = get_config_path() config = _sanitize_data(_parse_config(path)) diff --git a/content/pages/examples/flask/flask-blueprints-blueprint.markdown b/content/pages/examples/flask/flask-blueprints-blueprint.markdown index d74117ac5..16a0bb9c3 100644 --- a/content/pages/examples/flask/flask-blueprints-blueprint.markdown +++ b/content/pages/examples/flask/flask-blueprints-blueprint.markdown @@ -32,7 +32,7 @@ scenarios. CTFd is open sourced under the import base64 import requests -~~from flask import Blueprint +~~from flask import Blueprint, abort from flask import current_app as app from flask import redirect, render_template, request, session, url_for from itsdangerous.exc import BadSignature, BadTimeSignature, SignatureExpired @@ -309,7 +309,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) backend, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling and many others. flask-base is provided as open source under the @@ -649,7 +649,7 @@ def swagger_static(filename): ## Example 10 from Flask-WTF [Flask-WTF](https://github.com/lepture/flask-wtf) -([project documentation](https://flask-wtf.readthedocs.io/en/stable/) +([project documentation](https://flask-wtf.readthedocs.io/) and [PyPI page](https://pypi.org/project/Flask-WTF/)) provides a bridge between [Flask](/flask.html) and the the diff --git a/content/pages/examples/flask/flask-cli-dispatchingapp.markdown b/content/pages/examples/flask/flask-cli-dispatchingapp.markdown index facbdfdef..aa8140dd9 100644 --- a/content/pages/examples/flask/flask-cli-dispatchingapp.markdown +++ b/content/pages/examples/flask/flask-cli-dispatchingapp.markdown @@ -90,7 +90,7 @@ def _make_wsgi_app(info, url, evalex_whitelist, proxy): return info.load_app() url_data = url_parse(url) -~~ app = DispatchingApp(_load_app) +~~ app = DispatchingApp(_load_app, use_eager_loading=False) app = DebuggedIndico(app, evalex_whitelist) app = _make_indico_dispatcher(app, url_data.path) if proxy: diff --git a/content/pages/examples/flask/flask-cli-flaskgroup.markdown b/content/pages/examples/flask/flask-cli-flaskgroup.markdown index 5e87669b6..1b39ccb93 100644 --- a/content/pages/examples/flask/flask-cli-flaskgroup.markdown +++ b/content/pages/examples/flask/flask-cli-flaskgroup.markdown @@ -43,7 +43,6 @@ import traceback from datetime import datetime import click -import click_log from flask import current_app ~~from flask.cli import FlaskGroup, ScriptInfo, with_appcontext from flask_alembic import alembic_click @@ -52,20 +51,29 @@ from sqlalchemy_utils.functions import database_exists from werkzeug.utils import import_string from flaskbb import create_app -from flaskbb.cli.utils import (EmailType, FlaskBBCLIError, get_version, - prompt_config_path, prompt_save_user, - write_config) +from flaskbb.cli.utils import ( + EmailType, + FlaskBBCLIError, + get_version, + prompt_config_path, + prompt_save_user, + write_config, +) from flaskbb.extensions import alembic, celery, db, whooshee -from flaskbb.utils.populate import (create_default_groups, - create_default_settings, create_latest_db, - create_test_data, create_welcome_forum, - insert_bulk_data, run_plugin_migrations, - update_settings_from_fixture) +from flaskbb.utils.populate import ( + create_default_groups, + create_default_settings, + create_latest_db, + create_test_data, + create_welcome_forum, + insert_bulk_data, + run_plugin_migrations, + update_settings_from_fixture, +) from flaskbb.utils.translations import compile_translations logger = logging.getLogger(__name__) -click_log.basic_config(logger) ~~class FlaskBBGroup(FlaskGroup): @@ -83,8 +91,7 @@ click_log.basic_config(logger) self._loaded_flaskbb_plugins = True except Exception: logger.error( - "Error while loading CLI Plugins", - exc_info=traceback.format_exc() + "Error while loading CLI Plugins", exc_info=traceback.format_exc() ) else: shell_context_processors = app.pluggy.hook.flaskbb_shell_context() @@ -93,6 +100,7 @@ click_log.basic_config(logger) def get_command(self, ctx, name): self._load_flaskbb_plugins(ctx) + return super(FlaskBBGroup, self).get_command(ctx, name) ## ... source file continues with no further FlaskGroup examples... diff --git a/content/pages/examples/flask/flask-cli-scriptinfo.markdown b/content/pages/examples/flask/flask-cli-scriptinfo.markdown index 3a17c827b..469bcd7b5 100644 --- a/content/pages/examples/flask/flask-cli-scriptinfo.markdown +++ b/content/pages/examples/flask/flask-cli-scriptinfo.markdown @@ -43,7 +43,6 @@ import traceback from datetime import datetime import click -import click_log from flask import current_app ~~from flask.cli import FlaskGroup, ScriptInfo, with_appcontext from flask_alembic import alembic_click @@ -52,20 +51,29 @@ from sqlalchemy_utils.functions import database_exists from werkzeug.utils import import_string from flaskbb import create_app -from flaskbb.cli.utils import (EmailType, FlaskBBCLIError, get_version, - prompt_config_path, prompt_save_user, - write_config) +from flaskbb.cli.utils import ( + EmailType, + FlaskBBCLIError, + get_version, + prompt_config_path, + prompt_save_user, + write_config, +) from flaskbb.extensions import alembic, celery, db, whooshee -from flaskbb.utils.populate import (create_default_groups, - create_default_settings, create_latest_db, - create_test_data, create_welcome_forum, - insert_bulk_data, run_plugin_migrations, - update_settings_from_fixture) +from flaskbb.utils.populate import ( + create_default_groups, + create_default_settings, + create_latest_db, + create_test_data, + create_welcome_forum, + insert_bulk_data, + run_plugin_migrations, + update_settings_from_fixture, +) from flaskbb.utils.translations import compile_translations logger = logging.getLogger(__name__) -click_log.basic_config(logger) class FlaskBBGroup(FlaskGroup): @@ -83,8 +91,7 @@ class FlaskBBGroup(FlaskGroup): self._loaded_flaskbb_plugins = True except Exception: logger.error( - "Error while loading CLI Plugins", - exc_info=traceback.format_exc() + "Error while loading CLI Plugins", exc_info=traceback.format_exc() ) else: shell_context_processors = app.pluggy.hook.flaskbb_shell_context() @@ -100,7 +107,12 @@ class FlaskBBGroup(FlaskGroup): return super(FlaskBBGroup, self).list_commands(ctx) -def make_app(script_info): +def make_app(): + ctx = click.get_current_context(silent=True) + script_info = None + if ctx is not None: + script_info = ctx.obj + config_file = getattr(script_info, "config_file", None) instance_path = getattr(script_info, "instance_path", None) return create_app(config_file, instance_path) @@ -114,28 +126,28 @@ def set_instance(ctx, param, value): ~~ ctx.ensure_object(ScriptInfo).instance_path = value -@click.group(cls=FlaskBBGroup, create_app=make_app, add_version_option=False, - invoke_without_command=True) -@click.option("--config", expose_value=False, callback=set_config, - required=False, is_flag=False, is_eager=True, metavar="CONFIG", - help="Specify the config to use either in dotted module " - "notation e.g. 'flaskbb.configs.default.DefaultConfig' " - "or by using a path like '/path/to/flaskbb.cfg'") -@click.option("--instance", expose_value=False, callback=set_instance, - required=False, is_flag=False, is_eager=True, metavar="PATH", - help="Specify the instance path to use. By default the folder " - "'instance' next to the package or module is assumed to " - "be the instance path.") -@click.option("--version", expose_value=False, callback=get_version, - is_flag=True, is_eager=True, help="Show the FlaskBB version.") -@click.pass_context -@click_log.simple_verbosity_option(logger) -def flaskbb(ctx): - if ctx.invoked_subcommand is None: - click.echo(ctx.get_help()) - - -flaskbb.add_command(alembic_click, "db") +@click.group( + cls=FlaskBBGroup, + create_app=make_app, + add_version_option=False, + invoke_without_command=True, +) +@click.option( + "--config", + expose_value=False, + callback=set_config, + required=False, + is_flag=False, + is_eager=True, + metavar="CONFIG", + help="Specify the config to use either in dotted module " + "notation e.g. 'flaskbb.configs.default.DefaultConfig' " + "or by using a path like '/path/to/flaskbb.cfg'", +) +@click.option( + "--instance", + expose_value=False, + callback=set_instance, ## ... source file continues with no further ScriptInfo examples... diff --git a/content/pages/examples/flask/flask-cli-with-appcontext.markdown b/content/pages/examples/flask/flask-cli-with-appcontext.markdown index bb89e469a..e2d1ab675 100644 --- a/content/pages/examples/flask/flask-cli-with-appcontext.markdown +++ b/content/pages/examples/flask/flask-cli-with-appcontext.markdown @@ -43,7 +43,6 @@ import traceback from datetime import datetime import click -import click_log from flask import current_app ~~from flask.cli import FlaskGroup, ScriptInfo, with_appcontext from flask_alembic import alembic_click @@ -52,35 +51,29 @@ from sqlalchemy_utils.functions import database_exists from werkzeug.utils import import_string from flaskbb import create_app -from flaskbb.cli.utils import (EmailType, FlaskBBCLIError, get_version, - prompt_config_path, prompt_save_user, - write_config) +from flaskbb.cli.utils import ( + EmailType, + FlaskBBCLIError, + get_version, + prompt_config_path, + prompt_save_user, + write_config, +) from flaskbb.extensions import alembic, celery, db, whooshee -from flaskbb.utils.populate import (create_default_groups, - create_default_settings, create_latest_db, - create_test_data, create_welcome_forum, - insert_bulk_data, run_plugin_migrations, - update_settings_from_fixture) -from flaskbb.utils.translations import compile_translations - - -logger = logging.getLogger(__name__) -click_log.basic_config(logger) - - -class FlaskBBGroup(FlaskGroup): - def __init__(self, *args, **kwargs): +from flaskbb.utils.populate import ( + create_default_groups, + create_default_settings, + create_latest_db, + create_test_data, + create_welcome_forum, + insert_bulk_data, + run_plugin_migrations, + update_settings_from_fixture, ## ... source file abbreviated to get to with_appcontext examples ... - "'instance' next to the package or module is assumed to " - "be the instance path.") -@click.option("--version", expose_value=False, callback=get_version, - is_flag=True, is_eager=True, help="Show the FlaskBB version.") -@click.pass_context -@click_log.simple_verbosity_option(logger) def flaskbb(ctx): if ctx.invoked_subcommand is None: click.echo(ctx.get_help()) @@ -90,23 +83,40 @@ flaskbb.add_command(alembic_click, "db") @flaskbb.command() -@click.option("--welcome", "-w", default=True, is_flag=True, - help="Disable the welcome forum.") -@click.option("--force", "-f", default=False, is_flag=True, - help="Doesn't ask for confirmation.") +@click.option( + "--welcome", "-w", default=True, is_flag=True, help="Disable the welcome forum." +) +@click.option( + "--force", "-f", default=False, is_flag=True, help="Doesn't ask for confirmation." +) @click.option("--username", "-u", help="The username of the user.") -@click.option("--email", "-e", type=EmailType(), - help="The email address of the user.") +@click.option("--email", "-e", type=EmailType(), help="The email address of the user.") @click.option("--password", "-p", help="The password of the user.") -@click.option("--no-plugins", "-n", default=False, is_flag=True, - help="Don't run the migrations for the default plugins.") +@click.option( + "--no-plugins", + "-n", + default=False, + is_flag=True, + help="Don't run the migrations for the default plugins.", +) ~~@with_appcontext def install(welcome, force, username, email, password, no_plugins): + if not current_app.config["CONFIG_PATH"]: + click.secho( + "[!] No 'flaskbb.cfg' config found. " + "You can generate a configuration file with 'flaskbb makeconfig'.", + fg="red", + ) + sys.exit(1) + click.secho("[+] Installing FlaskBB...", fg="cyan") if database_exists(db.engine.url): - if force or click.confirm(click.style( - "Existing database found. Do you want to delete the old one and " - "create a new one?", fg="magenta") + if force or click.confirm( + click.style( + "Existing database found. Do you want to delete the old one and " + "create a new one?", + fg="magenta", + ) ): db.drop_all() else: @@ -114,46 +124,35 @@ def install(welcome, force, username, email, password, no_plugins): create_latest_db() - click.secho("[+] Creating default settings...", fg="cyan") - create_default_groups() - create_default_settings() - - click.secho("[+] Creating admin user...", fg="cyan") - prompt_save_user(username, email, password, "admin") - - if welcome: - click.secho("[+] Creating welcome forum...", fg="cyan") - create_welcome_forum() - ## ... source file abbreviated to get to with_appcontext examples ... - - if fixture or all_latest: - try: - settings = import_string( - "flaskbb.fixtures.{}".format(fixture) - ) - settings = settings.fixture except ImportError: - raise FlaskBBCLIError("{} fixture is not available" - .format(fixture), fg="red") + raise FlaskBBCLIError( + "{} fixture is not available".format(fixture), fg="red" + ) click.secho("[+] Updating fixtures...", fg="cyan") count = update_settings_from_fixture( fixture=settings, overwrite_group=force, overwrite_setting=force ) - click.secho("[+] {settings} settings in {groups} setting groups " - "updated.".format(groups=len(count), settings=sum( - len(settings) for settings in count.values()) - ), fg="green") + click.secho( + "[+] {settings} settings in {groups} setting groups " + "updated.".format( + groups=len(count), + settings=sum(len(settings) for settings in count.values()), + ), + fg="green", + ) -@flaskbb.command("celery", add_help_option=False, - context_settings={"ignore_unknown_options": True, - "allow_extra_args": True}) +@flaskbb.command( + "celery", + add_help_option=False, + context_settings={"ignore_unknown_options": True, "allow_extra_args": True}, +) @click.pass_context ~~@with_appcontext def start_celery(ctx): @@ -164,6 +163,7 @@ def start_celery(ctx): ~~@with_appcontext def shell_command(): import code + banner = "Python %s on %s\nInstance Path: %s" % ( sys.version, sys.platform, @@ -190,19 +190,21 @@ def shell_command(): @flaskbb.command("urls", short_help="Show routes for the app.") -@click.option("--route", "-r", "order_by", flag_value="rule", default=True, - help="Order by route") -@click.option("--endpoint", "-e", "order_by", flag_value="endpoint", - help="Order by endpoint") -@click.option("--methods", "-m", "order_by", flag_value="methods", - help="Order by methods") +@click.option( + "--route", "-r", "order_by", flag_value="rule", default=True, help="Order by route" +) +@click.option( + "--endpoint", "-e", "order_by", flag_value="endpoint", help="Order by endpoint" +) +@click.option( + "--methods", "-m", "order_by", flag_value="methods", help="Order by methods" +) ~~@with_appcontext def list_urls(order_by): from flask import current_app rules = sorted( - current_app.url_map.iter_rules(), - key=lambda rule: getattr(rule, order_by) + current_app.url_map.iter_rules(), key=lambda rule: getattr(rule, order_by) ) max_rule_len = max(len(rule.rule) for rule in rules) @@ -216,11 +218,12 @@ def list_urls(order_by): column_header_len = max_rule_len + max_endpoint_len + max_method_len + 4 column_template = "{:<%s} {:<%s} {:<%s}" % ( - max_rule_len, max_endpoint_len, max_method_len + max_rule_len, + max_endpoint_len, + max_method_len, ) - click.secho(column_template.format("Route", "Endpoint", "Methods"), - fg="blue", bold=True) + click.secho( ## ... source file continues with no further with_appcontext examples... diff --git a/content/pages/examples/flask/flask-ctx-after-this-request.markdown b/content/pages/examples/flask/flask-ctx-after-this-request.markdown index ad395c4b1..f18d7b9b7 100644 --- a/content/pages/examples/flask/flask-ctx-after-this-request.markdown +++ b/content/pages/examples/flask/flask-ctx-after-this-request.markdown @@ -40,17 +40,18 @@ The Flask-Security-Too project is provided as open source under the # unified_signin.py import time +import typing as t from flask import current_app as app ~~from flask import after_this_request, request, session from flask_login import current_user from werkzeug.datastructures import MultiDict -from werkzeug.local import LocalProxy from wtforms import BooleanField, RadioField, StringField, SubmitField, validators from .confirmable import requires_confirmation from .decorators import anonymous_user_required, auth_required, unauth_csrf from .forms import Form, Required, get_form_field_label +from .proxies import _security, _datastore from .quart_compat import get_quart_status from .signals import us_profile_changed, us_security_token_sent from .twofactor import ( @@ -63,7 +64,7 @@ from .utils import ( SmsSenderFactory, base_render_json, check_and_get_token_status, - config_value, + config_value as cv, do_flash, find_user, get_identity_attributes, @@ -84,7 +85,7 @@ from .utils import ( token=self.passcode.data, totp_secret=self.totp_secret, user=self.user, - window=config_value("US_TOKEN_VALIDITY"), + window=cv("US_TOKEN_VALIDITY"), ): self.passcode.errors.append(get_message("INVALID_PASSWORD_CODE")[0]) return False @@ -127,24 +128,24 @@ def us_signin_send_code(): ## ... source file abbreviated to get to after_this_request examples ... + form.submit.data = True + if form.validate_on_submit(): remember_me = form.remember.data if "remember" in form else None - if config_value("TWO_FACTOR") and form.authn_via in config_value( - "US_MFA_REQUIRED" - ): + if cv("TWO_FACTOR") and form.authn_via in cv("US_MFA_REQUIRED"): if request.is_json and request.content_length: - tf_validity_token = request.get_json().get("tf_validity_token", None) + tf_validity_token = request.get_json().get( # type: ignore + "tf_validity_token", None + ) else: tf_validity_token = request.cookies.get("tf_validity", default=None) tf_validity_token_is_valid = tf_verify_validility_token( tf_validity_token, form.user.fs_uniquifier ) - if config_value("TWO_FACTOR_REQUIRED") or is_tf_setup(form.user): - if config_value("TWO_FACTOR_ALWAYS_VALIDATE") or ( - not tf_validity_token_is_valid - ): + if cv("TWO_FACTOR_REQUIRED") or is_tf_setup(form.user): + if cv("TWO_FACTOR_ALWAYS_VALIDATE") or (not tf_validity_token_is_valid): return tf_login( form.user, @@ -163,7 +164,7 @@ def us_signin_send_code(): code_methods = _compute_code_methods() if _security._want_json(request): payload = { - "available_methods": config_value("US_ENABLED_METHODS"), + "available_methods": cv("US_ENABLED_METHODS"), "code_methods": code_methods, "identity_attributes": get_identity_attributes(), } @@ -174,9 +175,9 @@ def us_signin_send_code(): form.passcode.data = None - if form.requires_confirmation and _security.requires_confirmation_error_view: + if form.requires_confirmation and cv("REQUIRES_CONFIRMATION_ERROR_VIEW"): do_flash(*get_message("CONFIRMATION_REQUIRED")) - return redirect(get_url(_security.requires_confirmation_error_view)) + return redirect(get_url(cv("REQUIRES_CONFIRMATION_ERROR_VIEW"))) ## ... source file abbreviated to get to after_this_request examples ... @@ -185,7 +186,7 @@ def us_signin_send_code(): if _security.redirect_behavior == "spa": return redirect( get_url( - _security.login_error_view, + cv("LOGIN_ERROR_VIEW"), qparams=user.get_redirect_qparams({c: m}), ) ) @@ -193,14 +194,14 @@ def us_signin_send_code(): return redirect(url_for_security("us_signin")) if ( - config_value("TWO_FACTOR") - and "email" in config_value("US_MFA_REQUIRED") - and (config_value("TWO_FACTOR_REQUIRED") or is_tf_setup(user)) + cv("TWO_FACTOR") + and "email" in cv("US_MFA_REQUIRED") + and (cv("TWO_FACTOR_REQUIRED") or is_tf_setup(user)) ): if _security.redirect_behavior == "spa": return redirect( get_url( - _security.login_error_view, + cv("LOGIN_ERROR_VIEW"), qparams=user.get_redirect_qparams({"tf_required": 1}), ) ) @@ -210,7 +211,7 @@ def us_signin_send_code(): ~~ after_this_request(view_commit) if _security.redirect_behavior == "spa": return redirect( - get_url(_security.post_login_view, qparams=user.get_redirect_qparams()) + get_url(cv("POST_LOGIN_VIEW"), qparams=user.get_redirect_qparams()) ) do_flash(*get_message("PASSWORDLESS_LOGIN_SUCCESSFUL")) @@ -218,11 +219,11 @@ def us_signin_send_code(): @auth_required( - lambda: config_value("API_ENABLED_METHODS"), - within=lambda: config_value("FRESHNESS"), - grace=lambda: config_value("FRESHNESS_GRACE_PERIOD"), + lambda: cv("API_ENABLED_METHODS"), + within=lambda: cv("FRESHNESS"), + grace=lambda: cv("FRESHNESS_GRACE_PERIOD"), ) -def us_setup(): +def us_setup() -> "ResponseValue": form_class = _security.us_setup_form if request.is_json: @@ -250,7 +251,7 @@ def us_setup(): if invalid: m, c = get_message("API_ERROR") if expired: - m, c = get_message("US_SETUP_EXPIRED", within=config_value("US_SETUP_WITHIN")) + m, c = get_message("US_SETUP_EXPIRED", within=cv("US_SETUP_WITHIN")) if invalid or expired: if _security._want_json(request): payload = json_error_response(errors=m) @@ -268,7 +269,7 @@ def us_setup(): _datastore.us_set(current_user, method, state["totp_secret"], phone) us_profile_changed.send( - app._get_current_object(), user=current_user, method=method + app._get_current_object(), user=current_user, method=method # type: ignore ) if _security._want_json(request): return base_render_json( @@ -281,12 +282,12 @@ def us_setup(): else: do_flash(*get_message("US_SETUP_SUCCESSFUL")) return redirect( - get_url(_security.us_post_setup_view) - or get_url(_security.post_login_view) + get_url(cv("US_POST_SETUP_VIEW")) or get_url(cv("POST_LOGIN_VIEW")) ) if _security._want_json(request): return base_render_json(form, include_user=False) + m, c = get_message("INVALID_PASSWORD_CODE") ## ... source file continues with no further after_this_request examples... diff --git a/content/pages/examples/flask/flask-ctx-has-app-context.markdown b/content/pages/examples/flask/flask-ctx-has-app-context.markdown index a29697b0f..d4a03d102 100644 --- a/content/pages/examples/flask/flask-ctx-has-app-context.markdown +++ b/content/pages/examples/flask/flask-ctx-has-app-context.markdown @@ -143,6 +143,7 @@ The code is open sourced under the import ast import re +from collections import Counter from contextlib import contextmanager from babel import negotiate_locale @@ -150,7 +151,7 @@ from babel.core import LOCALE_ALIASES, Locale from babel.messages.pofile import read_po from babel.support import NullTranslations ~~from flask import current_app, g, has_app_context, has_request_context, request, session -from flask_babelex import Babel, Domain, get_domain +from flask_babel import Babel, Domain, get_domain from flask_pluginengine import current_plugin from speaklater import is_lazy_string, make_lazy_string from werkzeug.utils import cached_property diff --git a/content/pages/examples/flask/flask-ctx-has-request-context.markdown b/content/pages/examples/flask/flask-ctx-has-request-context.markdown index 235184bfc..9fe1163e0 100644 --- a/content/pages/examples/flask/flask-ctx-has-request-context.markdown +++ b/content/pages/examples/flask/flask-ctx-has-request-context.markdown @@ -322,23 +322,10 @@ import warnings from pprint import pformat import yaml -~~from flask import current_app, has_request_context, request, session +~~from flask import has_request_context, request, session from indico.core.config import config -from indico.util.i18n import set_best_lang -from indico.web.util import get_request_info - - -try: - from raven import setup_logging - from raven.contrib.celery import register_logger_signal, register_signal - from raven.contrib.flask import Sentry - from raven.handlers.logging import SentryHandler -except ImportError: - Sentry = object # so we can subclass - has_sentry = False -else: - has_sentry = True +from indico.web.util import get_request_info, get_request_user class AddRequestIDFilter: @@ -349,7 +336,8 @@ class AddRequestIDFilter: class AddUserIDFilter: def filter(self, record): -~~ record.user_id = str(session.user.id) if has_request_context() and session and session.user else '-' +~~ user = get_request_user()[0] if has_request_context() else None + record.user_id = str(session.user.id) if user else '-' return True @@ -375,70 +363,6 @@ class BlacklistFilter(logging.Filter): return not any(x.filter(record) for x in self.filters) - -## ... source file abbreviated to get to has_request_context examples ... - - - if formatter.pop('append_request_info', False): - assert '()' not in formatter - formatter['()'] = RequestInfoFormatter - if config.DB_LOG: - data['loggers']['indico._db'] = {'level': 'DEBUG', 'propagate': False, 'handlers': ['_db']} - data['handlers']['_db'] = {'class': 'logging.handlers.SocketHandler', 'host': '127.0.0.1', 'port': 9020} - if config.CUSTOMIZATION_DEBUG and config.CUSTOMIZATION_DIR: - data['loggers'].setdefault('indico.customization', {})['level'] = 'DEBUG' - logging.config.dictConfig(data) - if config.SENTRY_DSN: - if not has_sentry: - raise Exception('`raven` must be installed to use sentry logging') - init_sentry(app) - - @classmethod - def get(cls, name=None): - if name is None: - name = 'indico' - elif name != 'indico' and not name.startswith('indico.'): - name = 'indico.' + name - return logging.getLogger(name) - - -class IndicoSentry(Sentry): - def get_user_info(self, request): -~~ if not has_request_context() or not session.user: - return None - return {'id': session.user.id, - 'email': session.user.email, - 'name': session.user.full_name} - - def before_request(self, *args, **kwargs): - super().before_request() -~~ if not has_request_context(): - return - self.client.extra_context({'Endpoint': str(request.url_rule.endpoint) if request.url_rule else None, - 'Request ID': request.id}) - self.client.tags_context({'locale': set_best_lang()}) - - -def init_sentry(app): - sentry = IndicoSentry(wrap_wsgi=False, register_signal=True, logging=False) - sentry.init_app(app) - handler = SentryHandler(sentry.client, level=getattr(logging, config.SENTRY_LOGGING_LEVEL)) - handler.addFilter(BlacklistFilter({'indico.flask', 'celery.redirected'})) - setup_logging(handler) - register_logger_signal(sentry.client) - register_signal(sentry.client) - - -def sentry_log_exception(): - try: - sentry = current_app.extensions['sentry'] - except KeyError: - return - sentry.captureException() - - - - ## ... source file continues with no further has_request_context examples... ``` diff --git a/content/pages/examples/flask/flask-example-projects-code.markdown b/content/pages/examples/flask/flask-example-projects-code.markdown index a4b3d973e..e8a469a5d 100644 --- a/content/pages/examples/flask/flask-example-projects-code.markdown +++ b/content/pages/examples/flask/flask-example-projects-code.markdown @@ -99,7 +99,7 @@ is configured in JSON. The code is provided as open source under the is an example application that ties together the [intTellInput.js](https://github.com/jackocnr/intl-tel-input) JavaScript plugin with the -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) form-handling +[Flask-WTF](https://flask-wtf.readthedocs.io/) form-handling library. flask-phone-input is provided as open source under the [MIT license](https://github.com/miguelgrinberg/flask-phone-input/blob/1a1c227c044474ce0fe133493d7f8b0fb8312409/LICENSE). diff --git a/content/pages/examples/flask/flask-extensions-plug-ins.markdown b/content/pages/examples/flask/flask-extensions-plug-ins.markdown index 9467b3f26..de0d41668 100644 --- a/content/pages/examples/flask/flask-extensions-plug-ins.markdown +++ b/content/pages/examples/flask/flask-extensions-plug-ins.markdown @@ -219,7 +219,7 @@ building on, and the source code is open source under the ### Flask-WTF [Flask-WTF](https://github.com/lepture/flask-wtf) -([project documentation](https://flask-wtf.readthedocs.io/en/stable/) +([project documentation](https://flask-wtf.readthedocs.io/) and [PyPI page](https://pypi.org/project/Flask-WTF/)) provides a bridge between [Flask](/flask.html) and the the diff --git a/content/pages/examples/flask/flask-globals-current-app.markdown b/content/pages/examples/flask/flask-globals-current-app.markdown index 8dcdeaf60..450633b87 100644 --- a/content/pages/examples/flask/flask-globals-current-app.markdown +++ b/content/pages/examples/flask/flask-globals-current-app.markdown @@ -103,130 +103,84 @@ forms, and internationalization support. Flask App Builder is provided under the [BSD 3-Clause "New" or "Revised" license](https://github.com/dpgaspar/Flask-AppBuilder/blob/master/LICENSE). -[**Flask AppBuilder / flask_appbuilder / menu.py**](https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/./menu.py) +[**Flask AppBuilder / flask_appbuilder / validators.py**](https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/./validators.py) ```python -# menu.py -from typing import List - -~~from flask import current_app, url_for -from flask_babel import gettext as __ - -from .api import BaseApi, expose -from .basemanager import BaseManager -from .security.decorators import permission_name, protect - - -class MenuItem(object): - def __init__(self, name, href="", icon="", label="", childs=None, baseview=None): - self.name = name - self.href = href - self.icon = icon - self.label = label - self.childs = childs or [] - self.baseview = baseview - - def get_url(self): - if not self.href: - if not self.baseview: - return "" - else: - return url_for(f"{self.baseview.endpoint}.{self.baseview.default_view}") - else: - try: +# validators.py +import re +from typing import Optional +~~from flask import current_app +from flask_appbuilder.exceptions import PasswordComplexityValidationError +from flask_appbuilder.models.base import BaseInterface +from flask_babel import gettext +from wtforms import Field, Form, ValidationError + +password_complexity_regex = re.compile( + r"""( + ^(?=.*[A-Z].*[A-Z]) # at least two capital letters + (?=.*[^0-9a-zA-Z]) # at least one of these special characters + (?=.*[0-9].*[0-9]) # at least two numeric digits + (?=.*[a-z].*[a-z].*[a-z]) # at least three lower case letters + .{10,} # at least 10 total characters + $ + )""", + re.VERBOSE, +) -## ... source file abbreviated to get to current_app examples ... +class Unique: - self.menu = [] - if reverse: - extra_classes = extra_classes + "navbar-inverse" - self.extra_classes = extra_classes + field_flags = ("unique",) - @property - def reverse(self): - return "navbar-inverse" in self.extra_classes - - def get_list(self): - return self.menu - - def get_flat_name_list(self, menu: "Menu" = None, result: List = None) -> List: - menu = menu or self.menu - result = result or [] - for item in menu: - result.append(item.name) - if item.childs: - result.extend(self.get_flat_name_list(menu=item.childs, result=result)) - return result + def __init__( + self, datamodel: BaseInterface, col_name: str, message: Optional[str] = None + ) -> None: + self.datamodel = datamodel + self.col_name = col_name + self.message = message - def get_data(self, menu=None): - menu = menu or self.menu - ret_list = [] - -~~ allowed_menus = current_app.appbuilder.sm.get_user_menu_access( - self.get_flat_name_list() + def __call__(self, form: Form, field: Field) -> None: + filters = self.datamodel.get_filters().add_filter( + self.col_name, self.datamodel.FilterEqual, field.data ) - - for i, item in enumerate(menu): - if item.name == "-" and not i == len(menu) - 1: - ret_list.append("-") - elif item.name not in allowed_menus: - continue - elif item.childs: - ret_list.append( - { - "name": item.name, - "icon": item.icon, - "label": __(str(item.label)), - "childs": self.get_data(menu=item.childs), - } - ) - else: - ret_list.append( - { - "name": item.name, - "icon": item.icon, - "label": __(str(item.label)), - "url": item.get_url(), - - -## ... source file abbreviated to get to current_app examples ... + count, obj = self.datamodel.query(filters) + if count > 0: + if not hasattr(form, "_id") or form._id != self.datamodel.get_keys(obj)[0]: + if self.message is None: + self.message = field.gettext(u"Already exists.") + raise ValidationError(self.message) - category=category, icon=category_icon, label=category_label - ) - new_menu_item = MenuItem( - name=name, href=href, icon=icon, label=label, baseview=baseview - ) - self.find(category).childs.append(new_menu_item) +class PasswordComplexityValidator: - def add_separator(self, category=""): - menu_item = self.find(category) - if menu_item: - menu_item.childs.append(MenuItem("-")) - else: - raise Exception( - "Menu separator does not have correct category {}".format(category) + def __call__(self, form: Form, field: Field) -> None: +~~ if current_app.config.get("FAB_PASSWORD_COMPLEXITY_ENABLED", False): +~~ password_complexity_validator = current_app.config.get( + "FAB_PASSWORD_COMPLEXITY_VALIDATOR", None ) - - -class MenuApi(BaseApi): - resource_name = "menu" - openapi_spec_tag = "Menu" - - @expose("/", methods=["GET"]) - @protect(allow_browser_login=True) - @permission_name("get") - def get_menu_data(self): -~~ return self.response(200, result=current_app.appbuilder.menu.get_data()) - - -class MenuApiManager(BaseManager): - def register_views(self): - if self.appbuilder.app.config.get("FAB_ADD_MENU_API", True): - self.appbuilder.add_api(MenuApi) - + if password_complexity_validator is not None: + try: + password_complexity_validator(field.data) + except PasswordComplexityValidationError as exc: + raise ValidationError(str(exc)) + else: + try: + default_password_complexity(field.data) + except PasswordComplexityValidationError as exc: + raise ValidationError(str(exc)) + + +def default_password_complexity(password: str) -> None: + match = re.search(password_complexity_regex, password) + if not match: + raise PasswordComplexityValidationError( + gettext( + "Must have at least two capital letters," + " one special character, two digits, three lower case letters and" + " a minimal length of 10." + ) + ) ## ... source file continues with no further current_app examples... @@ -630,7 +584,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) backend, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling and many others. flask-base is provided as open source under the @@ -1452,65 +1406,72 @@ Python code. It does this by adding a `{% meld_scripts %}` tag to the Flask template engine and then inserting components written in Python scripts created by a developer. -[**Flask-Meld / flask_meld / component.py**](https://github.com/mikeabrahamsen/Flask-Meld/blob/main/flask_meld/./component.py) +[**Flask-Meld / flask_meld / message.py**](https://github.com/mikeabrahamsen/Flask-Meld/blob/main/flask_meld/./message.py) ```python -# component.py -import os -import uuid -from importlib.util import module_from_spec, spec_from_file_location +# message.py +import ast +from werkzeug.wrappers.response import Response +import functools +from .component import get_component_class +~~from flask import jsonify, current_app import orjson -from bs4 import BeautifulSoup -from bs4.formatter import HTMLFormatter -~~from flask import render_template, current_app, jsonify -def convert_to_snake_case(s): - s.replace("-", "_") - return s +def process_message(message): + meld_id = message["id"] + component_name = message["componentName"] + action_queue = message["actionQueue"] + data = message["data"] + Component = get_component_class(component_name) + component = Component(meld_id, **data) + return_data = None -def convert_to_camel_case(s): - s = convert_to_snake_case(s) - return "".join(word.title() for word in s.split("_")) + for action in action_queue: + payload = action.get("payload", None) + if "syncInput" in action["type"]: + if hasattr(component, payload["name"]): + setattr(component, payload["name"], payload["value"]) + if component._form: + field_name = payload.get("name") + if field_name in component._form._fields: + field = getattr(component._form, field_name) + component._set_field_data(field_name, payload["value"]) + component.updated(field) -def get_component_class(component_name): - module_name = convert_to_snake_case(component_name) - class_name = convert_to_camel_case(module_name) - module = get_component_module(module_name) - component_class = getattr(module, class_name) +## ... source file abbreviated to get to current_app examples ... - return component_class -def get_component_module(module_name): -~~ user_specified_dir = current_app.config.get("MELD_COMPONENT_DIR", None) + if "(" in call_method_name and call_method_name.endswith(")"): + param_idx = call_method_name.index("(") + params_str = call_method_name[param_idx:] + + method_name = call_method_name.replace(params_str, "") + + params_str = params_str[1:-1] + if params_str != "": + try: + params = ast.literal_eval("[" + params_str + "]") + except (ValueError, SyntaxError): + params = list(map(str.strip, params_str.split(","))) + + return method_name, params + + +def listen(*event_names: str): + def dec(func): + func._meld_event_names = event_names + return func + return dec - if not user_specified_dir: - try: - name = getattr(current_app, "name", None) - full_path = os.path.join(name, "meld", "components", module_name + ".py") - module = load_module_from_path(full_path, module_name) - except FileNotFoundError: - full_path = os.path.join("meld", "components", module_name + ".py") - module = load_module_from_path(full_path, module_name) - return module - else: - try: - full_path = os.path.join(user_specified_dir, module_name + ".py") - module = load_module_from_path(full_path, module_name) - except FileNotFoundError: - full_path = os.path.join( - user_specified_dir, "components", module_name + ".py" - ) - module = load_module_from_path(full_path, module_name) - return module +def emit(event_name: str, **kwargs): +~~ current_app.socketio.emit("meld-event", {"event": event_name, "message": kwargs}) -def load_module_from_path(full_path, module_name): - spec = spec_from_file_location(module_name, full_path) ## ... source file continues with no further current_app examples... @@ -1894,7 +1855,7 @@ except ImportError: ## Example 14 from Flask-WTF [Flask-WTF](https://github.com/lepture/flask-wtf) -([project documentation](https://flask-wtf.readthedocs.io/en/stable/) +([project documentation](https://flask-wtf.readthedocs.io/) and [PyPI page](https://pypi.org/project/Flask-WTF/)) provides a bridge between [Flask](/flask.html) and the the @@ -2171,20 +2132,24 @@ The Flask-Security-Too project is provided as open source under the # core.py from datetime import datetime, timedelta +import re +import typing as t import warnings import pkg_resources -~~from flask import _request_ctx_stack, current_app, render_template +~~from flask import _request_ctx_stack, current_app +from flask.json import JSONEncoder from flask_login import AnonymousUserMixin, LoginManager from flask_login import UserMixin as BaseUserMixin from flask_login import current_user from flask_principal import Identity, Principal, RoleNeed, UserNeed, identity_loaded +from flask_wtf import FlaskForm from itsdangerous import URLSafeTimedSerializer from passlib.context import CryptContext from werkzeug.datastructures import ImmutableList from werkzeug.local import LocalProxy -from .babel import get_i18n_domain, have_babel +from .babel import FsDomain from .decorators import ( default_reauthn_handler, default_unauthn_handler, @@ -2197,63 +2162,6 @@ from .forms import ( LoginForm, PasswordlessLoginForm, RegisterForm, - ResetPasswordForm, - SendConfirmationForm, - - -## ... source file abbreviated to get to current_app examples ... - - - UnifiedSigninSetupForm, - UnifiedSigninSetupValidateForm, - UnifiedVerifyForm, - us_send_security_token, -) -from .totp import Totp -from .utils import _ -from .utils import config_value as cv -from .utils import ( - FsJsonEncoder, - FsPermNeed, - csrf_cookie_handler, - default_want_json, - get_config, - get_identity_attribute, - get_identity_attributes, - get_message, - localize_callback, - set_request_attr, - uia_email_mapper, - url_for_security, - verify_and_update_password, -) -from .views import create_blueprint, default_render_json - -~~_security = LocalProxy(lambda: current_app.extensions["security"]) -_datastore = LocalProxy(lambda: _security.datastore) - -AUTHN_MECHANISMS = ("basic", "session", "token") - - -_default_config = { - "BLUEPRINT_NAME": "security", - "CLI_ROLES_NAME": "roles", - "CLI_USERS_NAME": "users", - "URL_PREFIX": None, - "SUBDOMAIN": None, - "FLASH_MESSAGES": True, - "I18N_DOMAIN": "flask_security", - "I18N_DIRNAME": pkg_resources.resource_filename("flask_security", "translations"), - "EMAIL_VALIDATOR_ARGS": None, - "PASSWORD_HASH": "bcrypt", - "PASSWORD_SALT": None, - "PASSWORD_SINGLE_HASH": { - "django_argon2", - "django_bcrypt_sha256", - "django_pbkdf2_sha256", - "django_pbkdf2_sha1", - "django_bcrypt", - "django_salted_md5", ## ... source file abbreviated to get to current_app examples ... @@ -2277,7 +2185,7 @@ _default_config = { "TWO_FACTOR_VALIDITY_COOKIE": { "httponly": True, "secure": False, - "samesite": None, + "samesite": "Strict", }, "CONFIRM_EMAIL_WITHIN": "5 days", "RESET_PASSWORD_WITHIN": "5 days", @@ -2314,33 +2222,33 @@ _default_config = { ## ... source file abbreviated to get to current_app examples ... - - for key, value in _default_config.items(): - app.config.setdefault("SECURITY_" + key, value) - - for key, value in _default_messages.items(): - app.config.setdefault("SECURITY_MSG_" + key, value) + for key, value in get_config(app).items(): + setattr(self, key.lower(), value) identity_loaded.connect_via(app)(_on_identity_loaded) - self._state = state = _get_state(app, datastore, **kwargs) - if hasattr(datastore, "user_model") and not hasattr( - datastore.user_model, "fs_uniquifier" + if hasattr(self.datastore, "user_model") and not hasattr( + self.datastore.user_model, "fs_uniquifier" ): # pragma: no cover raise ValueError("User model must contain fs_uniquifier as of 4.0.0") - if register_blueprint: - bp = create_blueprint( - app, state, __name__, json_encoder=kwargs["json_encoder_cls"] - ) - app.register_blueprint(bp) - app.context_processor(_context_processor) + for uia in cv("USER_IDENTITY_ATTRIBUTES", app=app): # pragma: no cover + if not isinstance(uia, dict): + raise ValueError( + "SECURITY_USER_IDENTITY_ATTRIBUTES changed semantics" + " in 4.0 - please see release notes." + ) + if len(list(uia.keys())) != 1: + raise ValueError( + "Each element in SECURITY_USER_IDENTITY_ATTRIBUTES" + " must have one and only one key." + ) @app.before_first_request def _register_i18n(): if "_" not in app.jinja_env.globals: -~~ current_app.jinja_env.globals["_"] = state.i18n_domain.gettext -~~ current_app.jinja_env.globals["_fsdomain"] = state.i18n_domain.gettext +~~ current_app.jinja_env.globals["_"] = self.i18n_domain.gettext +~~ current_app.jinja_env.globals["_fsdomain"] = self.i18n_domain.gettext @app.before_first_request def _csrf_init(): @@ -2370,40 +2278,42 @@ _default_config = { ) csrf_cookie = cv("CSRF_COOKIE") - if csrf_cookie and csrf_cookie["key"] and not csrf: + if csrf_cookie and csrf_cookie.get("key", None): +~~ current_app.config["SECURITY_CSRF_COOKIE_NAME"] = csrf_cookie.pop("key") + if cv("CSRF_COOKIE_NAME") and not csrf: raise ValueError( "CSRF_COOKIE defined however CsrfProtect not part of application" ) if csrf: csrf.exempt("flask_security.views.logout") - if csrf_cookie and csrf_cookie["key"]: + if cv("CSRF_COOKIE_NAME"): ~~ current_app.after_request(csrf_cookie_handler) ~~ current_app.config["WTF_CSRF_HEADERS"].append(cv("CSRF_HEADER")) - @app.before_first_request - def check_babel(): - if have_babel() and "babel" not in app.extensions: - raise ValueError( - "Flask-Babel or Flask-BabelEx is installed but not initialized" - ) - - state._phone_util = state.phone_util_cls(app) - state._mail_util = state.mail_util_cls(app) - state._password_util = state.password_util_cls(app) - - app.extensions["security"] = state - - if hasattr(app, "cli"): - from .cli import users, roles - - if state.cli_users_name: - app.cli.add_command(users, state.cli_users_name) - if state.cli_roles_name: - app.cli.add_command(roles, state.cli_roles_name) - - for newc, oldc in [ - ("SECURITY_SMS_SERVICE", "SECURITY_TWO_FACTOR_SMS_SERVICE"), + self._phone_util = self.phone_util_cls(app) + self._mail_util = self.mail_util_cls(app) + self._password_util = self.password_util_cls(app) + self._username_util = self.username_util_cls(app) + rvre = cv("REDIRECT_VALIDATE_RE", app=app) + if rvre: + self._redirect_validate_re = re.compile(rvre) + + if not hasattr(app, "login_manager") or not self.login_manager: + self.login_manager = _get_login_manager(app, self.anonymous_user) + + self.remember_token_serializer = _get_serializer(app, "remember") + self.login_serializer = _get_serializer(app, "login") + self.reset_serializer = _get_serializer(app, "reset") + self.confirm_serializer = _get_serializer(app, "confirm") + self.us_setup_serializer = _get_serializer(app, "us_setup") + self.tf_validity_serializer = _get_serializer(app, "two_factor_validity") + self.principal = _get_principal(app) + self.pwd_context = _get_pwd_context(app) + self.hashing_context = _get_hashing_context(app) + self.i18n_domain = FsDomain(app) + + if cv("USERNAME_ENABLE", app): ## ... source file continues with no further current_app examples... diff --git a/content/pages/examples/flask/flask-globals-g.markdown b/content/pages/examples/flask/flask-globals-g.markdown index e84628965..6c47941e6 100644 --- a/content/pages/examples/flask/flask-globals-g.markdown +++ b/content/pages/examples/flask/flask-globals-g.markdown @@ -42,7 +42,7 @@ import datetime import json import logging import re -from typing import Dict, List, Set +from typing import Dict, List, Optional, Set, Tuple ~~from flask import g, session, url_for from flask_babel import lazy_gettext as _ @@ -186,12 +186,6 @@ from .views import ( raise NotImplementedError - def add_permission_view_menu(self, permission_name, view_menu_name): - raise NotImplementedError - - def del_permission_view_menu(self, permission_name, view_menu_name, cascade=True): - raise NotImplementedError - def exist_permission_on_views(self, lst, item): raise NotImplementedError @@ -204,6 +198,12 @@ from .views import ( def del_permission_role(self, role, perm_view): raise NotImplementedError + def export_roles(self, path: Optional[str] = None) -> None: + raise NotImplementedError + + def import_roles(self, path: str) -> None: + raise NotImplementedError + def load_user(self, pk): return self.get_user_by_id(int(pk)) @@ -679,7 +679,7 @@ class HTTPBasicAuth(HTTPAuth): ## Example 6 from Flask-WTF [Flask-WTF](https://github.com/lepture/flask-wtf) -([project documentation](https://flask-wtf.readthedocs.io/en/stable/) +([project documentation](https://flask-wtf.readthedocs.io/) and [PyPI page](https://pypi.org/project/Flask-WTF/)) provides a bridge between [Flask](/flask.html) and the the diff --git a/content/pages/examples/flask/flask-globals-request.markdown b/content/pages/examples/flask/flask-globals-request.markdown index 4368a1a2f..be097b278 100644 --- a/content/pages/examples/flask/flask-globals-request.markdown +++ b/content/pages/examples/flask/flask-globals-request.markdown @@ -207,14 +207,19 @@ def test_that_request_path_hijacking_works_properly(): def test_theme_fallback_config(): - app = create_ctfd() + + class ThemeFallbackConfig(TestingConfig): + THEME_FALLBACK = False + + app = create_ctfd(config=ThemeFallbackConfig) try: - os.mkdir(os.path.join(app.root_path, "themes", "foo")) + os.mkdir(os.path.join(app.root_path, "themes", "foo_fallback")) except OSError: pass with app.app_context(): - set_config("ctf_theme", "foo") + app.config["THEME_FALLBACK"] = False + set_config("ctf_theme", "foo_fallback") assert app.config["THEME_FALLBACK"] == False with app.test_client() as client: try: @@ -222,11 +227,6 @@ def test_theme_fallback_config(): except TemplateNotFound: pass try: - r = client.get("/themes/foo/static/js/pages/main.dev.js") - except TemplateNotFound: - pass - destroy_ctfd(app) - ## ... source file continues with no further request examples... @@ -658,7 +658,7 @@ class UnbanUser(MethodView): data.append( { "id": user.id, - "type": "unban", + "type": "ban", "reverse": "ban", "reverse_name": _("Ban"), "reverse_url": url_for("management.ban_user", @@ -1997,7 +1997,7 @@ class marshal_with_field(object): ## Example 12 from Flask-WTF [Flask-WTF](https://github.com/lepture/flask-wtf) -([project documentation](https://flask-wtf.readthedocs.io/en/stable/) +([project documentation](https://flask-wtf.readthedocs.io/) and [PyPI page](https://pypi.org/project/Flask-WTF/)) provides a bridge between [Flask](/flask.html) and the the @@ -2159,7 +2159,7 @@ def generate_csrf(secret_key=None, token_key=None): starter project to build a software-as-a-service (SaaS) web application in [Flask](/flask.html), with [Stripe](/stripe.html) for billing. The boilerplate relies on many common Flask extensions such as -[Flask-WTF](https://flask-wtf.readthedocs.io/en/latest/), +[Flask-WTF](https://flask-wtf.readthedocs.io/), [Flask-Login](https://flask-login.readthedocs.io/en/latest/), [Flask-Admin](https://flask-admin.readthedocs.io/en/latest/), and many others. The project is provided as open source under the @@ -2226,6 +2226,7 @@ The Flask-Security-Too project is provided as open source under the # forms.py import inspect +import typing as t ~~from flask import Markup, current_app, request from flask_login import current_user @@ -2242,25 +2243,25 @@ from wtforms import ( ValidationError, validators, ) -from wtforms.fields.html5 import EmailField + +try: # pragma: no cover + from wtforms.fields import EmailField +except ImportError: + from wtforms.fields.html5 import EmailField from wtforms.validators import StopValidation from .babel import is_lazy_string, make_lazy_string from .confirmable import requires_confirmation from .utils import ( - _, - _datastore, - config_value, - do_flash, ## ... source file abbreviated to get to request examples ... - -class RegisterFormMixin: submit = SubmitField(get_form_field_label("register")) + username: t.ClassVar[Field] + def to_dict(self, only_user): def is_field_and_user_attr(member): @@ -2313,22 +2314,22 @@ class ForgotPasswordForm(Form, UserEmailFormMixin): ## ... source file abbreviated to get to request examples ... - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - - def validate(self): - if not super().validate(): - return False - if not self.user.is_active: self.email.errors.append(get_message("DISABLED_ACCOUNT")[0]) return False return True +login_email_field = EmailField( + get_form_field_label("email"), validators=[email_required] +) + +login_string_field = StringField( + get_form_field_label("email"), validators=[email_required] +) + + class LoginForm(Form, NextFormMixin): - email = EmailField(get_form_field_label("email"), validators=[email_required]) password = PasswordField( get_form_field_label("password"), validators=[password_required] ) @@ -2339,7 +2340,7 @@ class LoginForm(Form, NextFormMixin): super().__init__(*args, **kwargs) if not self.next.data: ~~ self.next.data = request.args.get("next", "") - self.remember.default = config_value("DEFAULT_REMEMBER_ME") + self.remember.default = cv("DEFAULT_REMEMBER_ME") if ( current_app.extensions["security"].recoverable and not self.password.description @@ -2382,7 +2383,7 @@ class RegisterForm(ConfirmRegisterForm, NextFormMixin): def validate(self): if not super().validate(): return False - if not config_value("UNIFIED_SIGNIN"): + if not cv("UNIFIED_SIGNIN"): if not self.password_confirm.data or not self.password_confirm.data.strip(): self.password_confirm.errors.append( get_message("PASSWORD_NOT_PROVIDED")[0] @@ -2949,6 +2950,7 @@ import configparser import json import logging import os +from typing import Union import requests import requests_cache @@ -2974,6 +2976,8 @@ requests_cache.install_cache(cache_name='news_cache', expire_after=300) APP = Flask(__name__) +SESSION = requests.Session() +SESSION.headers.update({'Authorization': API_KEY}) @APP.route('/', methods=['GET', 'POST']) @@ -2998,14 +3002,13 @@ def category(category): country = get_cookie('country') if country is not None: params.update({'country': country}) - response = requests.get(TOP_HEADLINES, - params=params, - headers={'Authorization': API_KEY}) + response = SESSION.get(TOP_HEADLINES, params=params) if response.status_code == 200: pages = count_pages(response.json()) if page > pages: page = pages - return redirect(url_for('category', category=category, page=page)) + return redirect( + url_for('category', category=category, page=page)) articles = parse_articles(response.json()) return render(articles, page, pages, country, category) elif response.status_code == 401: @@ -3014,7 +3017,7 @@ def category(category): @APP.route('/search/', methods=['GET', 'POST']) -def search(query): +def search(query: str): ~~ page = request.args.get('page', default=1, type=int) if page < 1: return redirect(url_for('search', query=query, page=1)) @@ -3026,9 +3029,7 @@ def search(query): } ~~ if request.method == 'POST': return do_post(page, category='search', current_query=query) - response = requests.get(EVERYTHING, - params=params, - headers={'Authorization': API_KEY}) + response = SESSION.get(EVERYTHING, params=params) pages = count_pages(response.json()) if page > pages: page = pages @@ -3062,7 +3063,7 @@ def do_post(page, category='general', current_query=None): return redirect(url_for('category', category=category, page=page)) -def parse_articles(response): +def parse_articles(response: dict) -> list: parsed_articles = [] if response.get('status') == 'ok': for article in response.get('articles'): @@ -3075,16 +3076,16 @@ def parse_articles(response): ## ... source file abbreviated to get to request examples ... + 'source': article['source']['name'] }) return parsed_articles -def count_pages(response): - pages = 0 +def count_pages(response: dict) -> int: if response.get('status') == 'ok': - pages = (-(-response.get('totalResults', 0) // PAGE_SIZE)) - return pages + return (-(-response.get('totalResults', 0) // PAGE_SIZE)) + return 0 def render(articles, page, pages, country, category): @@ -3099,9 +3100,8 @@ def render(articles, page, pages, country, category): pages=pages) -def get_cookie(key): -~~ cookie_value = request.cookies.get(key) - return cookie_value +def get_cookie(key: str) -> Union[str, None]: +~~ return request.cookies.get(key) if __name__ == '__main__': @@ -3502,6 +3502,8 @@ from core.db import Database import os import sys import platform +import urllib +import requests from multiprocessing import Process trape = core.stats.trape @@ -3512,13 +3514,11 @@ db = Database() class victim_server(object): @app.route("/" + trape.victim_path) def homeVictim(): - opener = urllib2.build_opener() - headers = victim_headers(request.user_agent) - opener.addheaders = headers + r = requests.get(trape.url_to_clone, headers=victim_headers2(request.user_agent)) if (trape.type_lure == 'local'): html = assignScripts(victim_inject_code(render_template("/" + trape.url_to_clone), 'payload', '/', trape.gmaps, trape.ipinfo)) else: - html = assignScripts(victim_inject_code(opener.open(trape.url_to_clone).read(), 'payload', trape.url_to_clone, trape.gmaps, trape.ipinfo)) + html = assignScripts(victim_inject_code(r.content, 'payload', trape.url_to_clone, trape.gmaps, trape.ipinfo)) return html @app.route("/register", methods=["POST"]) @@ -3611,7 +3611,7 @@ class victim_server(object): ~~ url = request.args.get('url') if url[0:4] != 'http': url = 'http://' + url - opener = urllib2.build_opener() + opener = urllib.request.build_opener() headers = victim_headers(request.user_agent) opener.addheaders = headers html = assignScripts(victim_inject_code(opener.open(url).read(), 'vscript', url, trape.gmaps, trape.ipinfo)) diff --git a/content/pages/examples/flask/flask-globals-session.markdown b/content/pages/examples/flask/flask-globals-session.markdown index 6659b6dbf..baf253747 100644 --- a/content/pages/examples/flask/flask-globals-session.markdown +++ b/content/pages/examples/flask/flask-globals-session.markdown @@ -41,7 +41,7 @@ scenarios. CTFd is open sourced under the import base64 import requests -from flask import Blueprint +from flask import Blueprint, abort from flask import current_app as app ~~from flask import redirect, render_template, request, session, url_for from itsdangerous.exc import BadSignature, BadTimeSignature, SignatureExpired @@ -73,13 +73,6 @@ auth = Blueprint("auth", __name__) ## ... source file abbreviated to get to session examples ... - ) - db.session.close() - - if is_teams_mode(): - return redirect(url_for("teams.private")) - - return redirect(url_for("challenges.listing")) else: return render_template("register.html", errors=errors) @@ -97,6 +90,13 @@ def login(): user = Users.query.filter_by(name=name).first() if user: + if user.password is None: + errors.append( + "Your account was registered with a 3rd party authentication provider. " + "Please try logging in with a configured authentication provider." + ) + return render_template("login.html", errors=errors) + if user and verify_password(request.form["password"], user.password): ~~ session.regenerate() @@ -271,6 +271,7 @@ from babel.core import get_locale_identifier from babel.dates import format_date as babel_format_date from babel.dates import format_datetime as babel_format_datetime from babel.dates import format_timedelta as babel_format_timedelta +from babel.dates import format_time as babel_format_time ~~from flask import current_app, flash, g, redirect, request, session, url_for from flask_allows import Permission from flask_babelplus import lazy_gettext as _ @@ -730,7 +731,7 @@ def login_required(func): ## Example 7 from Flask-WTF [Flask-WTF](https://github.com/lepture/flask-wtf) -([project documentation](https://flask-wtf.readthedocs.io/en/stable/) +([project documentation](https://flask-wtf.readthedocs.io/) and [PyPI page](https://pypi.org/project/Flask-WTF/)) provides a bridge between [Flask](/flask.html) and the the @@ -845,7 +846,7 @@ def _get_config( starter project to build a software-as-a-service (SaaS) web application in [Flask](/flask.html), with [Stripe](/stripe.html) for billing. The boilerplate relies on many common Flask extensions such as -[Flask-WTF](https://flask-wtf.readthedocs.io/en/latest/), +[Flask-WTF](https://flask-wtf.readthedocs.io/), [Flask-Login](https://flask-login.readthedocs.io/en/latest/), [Flask-Admin](https://flask-admin.readthedocs.io/en/latest/), and many others. The project is provided as open source under the @@ -932,10 +933,12 @@ The Flask-Security-Too project is provided as open source under the ```python # twofactor.py +import typing as t + ~~from flask import current_app as app, redirect, request, session from werkzeug.datastructures import MultiDict -from werkzeug.local import LocalProxy +from .proxies import _security, _datastore from .utils import ( SmsSenderFactory, base_render_json, @@ -955,8 +958,8 @@ from .signals import ( tf_profile_changed, ) -_security = LocalProxy(lambda: app.extensions["security"]) -_datastore = LocalProxy(lambda: _security.datastore) +if t.TYPE_CHECKING: # pragma: no cover + from flask import Response def tf_clean_session(): @@ -1213,130 +1216,150 @@ is a [Flask](/flask.html)-based web app for event management. The code is open sourced under the [MIT license](https://github.com/indico/indico/blob/master/LICENSE). -[**indico / indico / core / logger.py**](https://github.com/indico/indico/blob/master/indico/core/logger.py) +[**indico / indico / util / i18n.py**](https://github.com/indico/indico/blob/master/indico/util/i18n.py) ```python -# logger.py +# i18n.py -import logging -import logging.config -import logging.handlers -import os -import warnings -from pprint import pformat - -import yaml -~~from flask import current_app, has_request_context, request, session +import ast +import re +from collections import Counter +from contextlib import contextmanager + +from babel import negotiate_locale +from babel.core import LOCALE_ALIASES, Locale +from babel.messages.pofile import read_po +from babel.support import NullTranslations +~~from flask import current_app, g, has_app_context, has_request_context, request, session +from flask_babel import Babel, Domain, get_domain +from flask_pluginengine import current_plugin +from speaklater import is_lazy_string, make_lazy_string +from werkzeug.utils import cached_property from indico.core.config import config -from indico.util.i18n import set_best_lang -from indico.web.util import get_request_info +from indico.util.caching import memoize_request + +LOCALE_ALIASES = dict(LOCALE_ALIASES, en='en_GB') +RE_TR_FUNCTION = re.compile(r'''_\("([^"]*)"\)|_\('([^']*)'\)''', re.DOTALL | re.MULTILINE) -try: - from raven import setup_logging - from raven.contrib.celery import register_logger_signal, register_signal - from raven.contrib.flask import Sentry - from raven.handlers.logging import SentryHandler -except ImportError: - Sentry = object # so we can subclass - has_sentry = False -else: - has_sentry = True +babel = Babel() +_use_context = object() -class AddRequestIDFilter: - def filter(self, record): - record.request_id = request.id if has_request_context() else '0' * 16 - return True +def get_translation_domain(plugin_name=_use_context): + if plugin_name is None: + return get_domain() + else: + plugin = None + if has_app_context(): + from indico.core.plugins import plugin_engine + plugin = plugin_engine.get_plugin(plugin_name) if plugin_name is not _use_context else current_plugin -class AddUserIDFilter: - def filter(self, record): -~~ record.user_id = str(session.user.id) if has_request_context() and session and session.user else '-' - return True +## ... source file abbreviated to get to session examples ... -class RequestInfoFormatter(logging.Formatter): - def format(self, record): - rv = super().format(record) - info = get_request_info() - if info: - rv += '\n\n' + pformat(info) - return rv + def weekday(self, daynum, short=True): + return self.days['format']['abbreviated' if short else 'wide'][daynum] -class FormattedSubjectSMTPHandler(logging.handlers.SMTPHandler): - def getSubject(self, record): - return self.subject % record.__dict__ + @cached_property + def time_formats(self): + formats = super().time_formats + for k, v in formats.items(): + v.format = v.format.replace(':%(ss)s', '') + return formats -class BlacklistFilter(logging.Filter): - def __init__(self, names): - self.filters = [logging.Filter(name) for name in names] +def _remove_locale_script(locale): + parts = locale.split('_') # e.g. `en_GB` or `zh_Hans_CN` + return f'{parts[0]}_{parts[-1]}' - def filter(self, record): - return not any(x.filter(record) for x in self.filters) +@babel.localeselector +def set_best_lang(check_session=True): + from indico.core.config import config + if not has_request_context(): + return 'en_GB' if current_app.config['TESTING'] else config.DEFAULT_LOCALE + elif 'lang' in g: + return g.lang +~~ elif check_session and session.lang is not None: +~~ return session.lang -## ... source file abbreviated to get to session examples ... + all_locales = {_remove_locale_script(loc).lower(): loc for loc in get_all_locales()} + preferred = [x.replace('-', '_') for x in request.accept_languages.values()] + resolved_lang = negotiate_locale(preferred, list(all_locales), aliases=LOCALE_ALIASES) + + if not resolved_lang: + if current_app.config['TESTING']: + return 'en_GB' + + resolved_lang = config.DEFAULT_LOCALE - if formatter.pop('append_request_info', False): - assert '()' not in formatter - formatter['()'] = RequestInfoFormatter - if config.DB_LOG: - data['loggers']['indico._db'] = {'level': 'DEBUG', 'propagate': False, 'handlers': ['_db']} - data['handlers']['_db'] = {'class': 'logging.handlers.SocketHandler', 'host': '127.0.0.1', 'port': 9020} - if config.CUSTOMIZATION_DEBUG and config.CUSTOMIZATION_DIR: - data['loggers'].setdefault('indico.customization', {})['level'] = 'DEBUG' - logging.config.dictConfig(data) - if config.SENTRY_DSN: - if not has_sentry: - raise Exception('`raven` must be installed to use sentry logging') - init_sentry(app) - - @classmethod - def get(cls, name=None): - if name is None: - name = 'indico' - elif name != 'indico' and not name.startswith('indico.'): - name = 'indico.' + name - return logging.getLogger(name) - - -class IndicoSentry(Sentry): - def get_user_info(self, request): -~~ if not has_request_context() or not session.user: - return None -~~ return {'id': session.user.id, -~~ 'email': session.user.email, -~~ 'name': session.user.full_name} - - def before_request(self, *args, **kwargs): - super().before_request() - if not has_request_context(): - return - self.client.extra_context({'Endpoint': str(request.url_rule.endpoint) if request.url_rule else None, - 'Request ID': request.id}) - self.client.tags_context({'locale': set_best_lang()}) - - -def init_sentry(app): - sentry = IndicoSentry(wrap_wsgi=False, register_signal=True, logging=False) - sentry.init_app(app) - handler = SentryHandler(sentry.client, level=getattr(logging, config.SENTRY_LOGGING_LEVEL)) - handler.addFilter(BlacklistFilter({'indico.flask', 'celery.redirected'})) - setup_logging(handler) - register_logger_signal(sentry.client) - register_signal(sentry.client) - - -def sentry_log_exception(): try: - sentry = current_app.extensions['sentry'] + resolved_lang = all_locales[resolved_lang.lower()] except KeyError: + return 'en_GB' + + resolved_lang = re.sub(r'^([a-zA-Z]+)_([a-zA-Z]+)$', + lambda m: f'{m.group(1).lower()}_{m.group(2).upper()}', + resolved_lang) + + g.lang = resolved_lang + return resolved_lang + + +@memoize_request +def get_current_locale(): + return IndicoLocale.parse(set_best_lang()) + + +def get_all_locales(): + if babel.app is None: + return {} + else: + missing = object() + languages = {str(t): config.CUSTOM_LANGUAGES.get(str(t), (t.language_name.title(), t.territory_name)) + for t in babel.list_translations() + if config.CUSTOM_LANGUAGES.get(str(t), missing) is not None} + counts = Counter(x[0] for x in languages.values()) + return {code: (name, territory, counts[name] > 1) for code, (name, territory) in languages.items()} + + +def set_session_lang(lang): +~~ session.lang = lang + + +@contextmanager +def session_language(lang): +~~ old_lang = session.lang + + set_session_lang(lang) + yield + set_session_lang(old_lang) + + +def parse_locale(locale): + return IndicoLocale.parse(locale) + + +def extract_node(node, keywords, commentTags, options, parents=[None]): + if isinstance(node, ast.Str) and isinstance(parents[-1], (ast.Assign, ast.Call)): + matches = RE_TR_FUNCTION.findall(node.s) + for m in matches: + line = m[0] or m[1] + yield (node.lineno, '', line.split('\n'), ['old style recursive strings']) + else: + for cnode in ast.iter_child_nodes(node): + yield from extract_node(cnode, keywords, commentTags, options, parents=(parents + [node])) + + +def po_to_json(po_file, locale=None, domain=None): + with open(po_file, 'rb') as f: + po_data = read_po(f, locale=locale, domain=domain) ## ... source file continues with no further session examples... @@ -1473,7 +1496,7 @@ from threading import Lock from flask_socketio import SocketIO, emit, join_room, rooms, disconnect import core.stats import core.user -from user_objects import attacks_hook_message +from core.user_objects import attacks_hook_message from core.utils import utils from core.db import Database import sys diff --git a/content/pages/examples/flask/flask-helpers-flash.markdown b/content/pages/examples/flask/flask-helpers-flash.markdown index e89f89b9f..346bb3f6b 100644 --- a/content/pages/examples/flask/flask-helpers-flash.markdown +++ b/content/pages/examples/flask/flask-helpers-flash.markdown @@ -1462,12 +1462,12 @@ class ImportRoleMembersMixin: self.role.members |= users for user in new_members: self.logger.info(f'User {user} added to role {self.role} by {session.user}') -~~ flash(ngettext("{} member has been imported.", - "{} members have been imported.", +~~ flash(ngettext('{} member has been imported.', + '{} members have been imported.', len(users)).format(len(users)), 'success') if unknown_emails: -~~ flash(ngettext("There is no user with this email address: {}", - "There are no users with these email addresses: {}", +~~ flash(ngettext('There is no user with this email address: {}', + 'There are no users with these email addresses: {}', len(unknown_emails)).format(', '.join(unknown_emails)), 'warning') tpl = get_template_module('events/roles/_roles.html') return jsonify_data(html=tpl.render_role(self.role, collapsed=False, email_button=False)) diff --git a/content/pages/examples/flask/flask-helpers-get-root-path.markdown b/content/pages/examples/flask/flask-helpers-get-root-path.markdown index 53da00788..3b75f3a3e 100644 --- a/content/pages/examples/flask/flask-helpers-get-root-path.markdown +++ b/content/pages/examples/flask/flask-helpers-get-root-path.markdown @@ -37,6 +37,7 @@ import os import re import shutil import socket +import subprocess import sys from operator import attrgetter from pathlib import Path @@ -45,6 +46,8 @@ from smtplib import SMTP import click from click import wrap_text ~~from flask.helpers import get_root_path +from packaging.specifiers import SpecifierSet +from packaging.version import Version from pkg_resources import iter_entry_points from prompt_toolkit import prompt from prompt_toolkit.completion import PathCompleter, WordCompleter @@ -57,6 +60,7 @@ from sqlalchemy.pool import NullPool from terminaltables import AsciiTable from werkzeug.urls import url_parse +import indico from indico.core.db.sqlalchemy.util.models import import_all_models from indico.util.console import cformat from indico.util.string import validate_email @@ -66,9 +70,6 @@ def _echo(msg=''): click.echo(msg, err=True) -def _warn(msg): - msg = wrap_text(msg) - click.echo(click.style(msg, fg='yellow'), err=True) ## ... source file abbreviated to get to get_root_path examples ... @@ -129,14 +130,14 @@ def _prompt(message, default='', path=False, list_=None, required=True, validate ## ... source file abbreviated to get to get_root_path examples ... + 'SMTP_USE_CELERY = False' + ] - if dev: + if not self.system_notices: config_data += [ '', - '# Development options', - 'DB_LOG = True', - 'DEBUG = True', - 'SMTP_USE_CELERY = False' + '# Disable system notices', + 'SYSTEM_NOTICES_URL = None' ] config = '\n'.join(x for x in config_data if x is not None) @@ -173,11 +174,7 @@ def _prompt(message, default='', path=False, list_=None, required=True, validate _echo(cformat('Run %{green!}export INDICO_CONFIG={}%{reset} to use your config file') .format(self.config_path)) - if self.old_archive_dir: - _echo(cformat('Check %{green!}https://git.io/vHP6o%{reset} for a guide on how to ' - 'import data from Indico v1.2')) - else: - _echo(cformat('You can now run %{green!}indico db prepare%{reset} to initialize your Indico database')) + _echo(cformat('You can now run %{green!}indico db prepare%{reset} to initialize your Indico database')) diff --git a/content/pages/examples/flask/flask-helpers-make-response.markdown b/content/pages/examples/flask/flask-helpers-make-response.markdown index 64df32c79..877d38532 100644 --- a/content/pages/examples/flask/flask-helpers-make-response.markdown +++ b/content/pages/examples/flask/flask-helpers-make-response.markdown @@ -326,18 +326,17 @@ The Flask-Security-Too project is provided as open source under the from functools import partial import time +import typing as t from flask import ( Blueprint, after_this_request, - current_app, jsonify, request, session, ) from flask_login import current_user -from werkzeug.datastructures import MultiDict -from werkzeug.local import LocalProxy +from werkzeug.datastructures import ImmutableMultiDict, MultiDict from .changeable import change_user_password from .confirmable import ( @@ -347,6 +346,7 @@ from .confirmable import ( ) from .decorators import anonymous_user_required, auth_required, unauth_csrf from .passwordless import login_token_status, send_login_instructions +from .proxies import _security, _datastore from .quart_compat import get_quart_status from .unified_signin import ( us_signin, @@ -366,7 +366,7 @@ from .recoverable import ( from .utils import ( base_render_json, check_and_update_authn_fresh, - config_value, + config_value as cv, do_flash, get_message, get_post_login_redirect, @@ -390,9 +390,8 @@ if get_quart_status(): # pragma: no cover else: ~~ from flask import make_response, redirect - -_security = LocalProxy(lambda: current_app.extensions["security"]) -_datastore = LocalProxy(lambda: _security.datastore) +if t.TYPE_CHECKING: # pragma: no cover + from flask.typing import ResponseValue def default_render_json(payload, code, headers, user): @@ -408,7 +407,7 @@ def _ctx(endpoint): @unauth_csrf(fall_through=True) -def login(): +def login() -> "ResponseValue": if current_user.is_authenticated and request.method == "POST": @@ -418,7 +417,7 @@ def login(): ) return _security._render_json(payload, 400, None, None) else: - return redirect(get_url(_security.post_login_view)) + return redirect(get_url(cv("POST_LOGIN_VIEW"))) form_class = _security.login_form @@ -448,6 +447,7 @@ import configparser import json import logging import os +from typing import Union import requests import requests_cache @@ -473,20 +473,20 @@ requests_cache.install_cache(cache_name='news_cache', expire_after=300) APP = Flask(__name__) +SESSION = requests.Session() +SESSION.headers.update({'Authorization': API_KEY}) -@APP.route('/', methods=['GET', 'POST']) -def root(): ## ... source file abbreviated to get to make_response examples ... + 'pageSize': PAGE_SIZE + } if request.method == 'POST': return do_post(page, category='search', current_query=query) - response = requests.get(EVERYTHING, - params=params, - headers={'Authorization': API_KEY}) + response = SESSION.get(EVERYTHING, params=params) pages = count_pages(response.json()) if page > pages: page = pages @@ -520,7 +520,7 @@ def do_post(page, category='general', current_query=None): return redirect(url_for('category', category=category, page=page)) -def parse_articles(response): +def parse_articles(response: dict) -> list: parsed_articles = [] if response.get('status') == 'ok': for article in response.get('articles'): diff --git a/content/pages/examples/flask/flask-helpers-safe-join.markdown b/content/pages/examples/flask/flask-helpers-safe-join.markdown index 9df182563..1382094d8 100644 --- a/content/pages/examples/flask/flask-helpers-safe-join.markdown +++ b/content/pages/examples/flask/flask-helpers-safe-join.markdown @@ -60,7 +60,7 @@ from CTFd.utils.migrations import create_database, migrations, stamp_latest_revi from CTFd.utils.sessions import CachingSessionInterface from CTFd.utils.updates import update_check -__version__ = "3.3.0" +__version__ = "3.4.0" __channel__ = "oss" diff --git a/content/pages/examples/flask/flask-helpers-send-file.markdown b/content/pages/examples/flask/flask-helpers-send-file.markdown index 1676d0fa2..726a0993c 100644 --- a/content/pages/examples/flask/flask-helpers-send-file.markdown +++ b/content/pages/examples/flask/flask-helpers-send-file.markdown @@ -45,6 +45,7 @@ from flask import Blueprint, abort from flask import current_app as app ~~from flask import redirect, render_template, request, send_file, session, url_for from flask.helpers import safe_join +from jinja2.exceptions import TemplateNotFound from sqlalchemy.exc import IntegrityError from CTFd.cache import cache @@ -67,7 +68,6 @@ from CTFd.models import ( db, ) from CTFd.utils import config, get_config, set_config -from CTFd.utils import user as current_user ## ... source file abbreviated to get to send_file examples ... diff --git a/content/pages/examples/flask/flask-helpers-url-for.markdown b/content/pages/examples/flask/flask-helpers-url-for.markdown index a82f0f96d..9dab48b41 100644 --- a/content/pages/examples/flask/flask-helpers-url-for.markdown +++ b/content/pages/examples/flask/flask-helpers-url-for.markdown @@ -224,13 +224,19 @@ from .security.decorators import permission_name, protect class MenuItem(object): - def __init__(self, name, href="", icon="", label="", childs=None, baseview=None): + def __init__( + self, name, href="", icon="", label="", childs=None, baseview=None, cond=None + ): self.name = name self.href = href self.icon = icon self.label = label self.childs = childs or [] self.baseview = baseview + self.cond = cond + + def should_render(self) -> bool: + return bool(self.cond()) if self.cond is not None else True def get_url(self): if not self.href: @@ -291,7 +297,7 @@ import re import mistune ~~from flask import url_for -from jinja2 import Markup +from markupsafe import Markup from pluggy import HookimplMarker from pygments import highlight from pygments.formatters import HtmlFormatter @@ -350,7 +356,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) backend, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling and many others. flask-base is provided as open source under the @@ -759,7 +765,7 @@ def ui_for(api): starter project to build a software-as-a-service (SaaS) web application in [Flask](/flask.html), with [Stripe](/stripe.html) for billing. The boilerplate relies on many common Flask extensions such as -[Flask-WTF](https://flask-wtf.readthedocs.io/en/latest/), +[Flask-WTF](https://flask-wtf.readthedocs.io/), [Flask-Login](https://flask-login.readthedocs.io/en/latest/), [Flask-Admin](https://flask-admin.readthedocs.io/en/latest/), and many others. The project is provided as open source under the diff --git a/content/pages/examples/flask/flask-json-jsonencoder.markdown b/content/pages/examples/flask/flask-json-jsonencoder.markdown index c1bb443db..7c217d86c 100644 --- a/content/pages/examples/flask/flask-json-jsonencoder.markdown +++ b/content/pages/examples/flask/flask-json-jsonencoder.markdown @@ -32,122 +32,43 @@ can work securely with Flask-based backends and that guidance by the The Flask-Security-Too project is provided as open source under the [MIT license](https://github.com/Flask-Middleware/flask-security/blob/master/LICENSE). -[**Flask-Security-Too / flask_security / utils.py**](https://github.com/Flask-Middleware/flask-security/blob/master/flask_security/./utils.py) +[**Flask-Security-Too / flask_security / core.py**](https://github.com/Flask-Middleware/flask-security/blob/master/flask_security/./core.py) ```python -# utils.py -import abc -import base64 -import datetime -from functools import partial -import hashlib -import hmac -import time -from typing import Dict, List +# core.py + +from datetime import datetime, timedelta +import re +import typing as t import warnings -from datetime import timedelta -from urllib.parse import parse_qsl, parse_qs, urlsplit, urlunsplit, urlencode -import urllib.request -import urllib.error -from flask import ( - _request_ctx_stack, - after_this_request, - current_app, - flash, - g, - request, - session, - url_for, -) +import pkg_resources +from flask import _request_ctx_stack, current_app ~~from flask.json import JSONEncoder -from flask_login import login_user as _login_user -from flask_login import logout_user as _logout_user +from flask_login import AnonymousUserMixin, LoginManager +from flask_login import UserMixin as BaseUserMixin from flask_login import current_user -from flask_login import COOKIE_NAME as REMEMBER_COOKIE_NAME -from flask_principal import AnonymousIdentity, Identity, identity_changed, Need -from flask_wtf import csrf -from wtforms import ValidationError -from itsdangerous import BadSignature, SignatureExpired +from flask_principal import Identity, Principal, RoleNeed, UserNeed, identity_loaded +from flask_wtf import FlaskForm +from itsdangerous import URLSafeTimedSerializer +from passlib.context import CryptContext +from werkzeug.datastructures import ImmutableList from werkzeug.local import LocalProxy -from werkzeug.datastructures import MultiDict - -from .quart_compat import best, get_quart_status -from .signals import user_authenticated - -_security = LocalProxy(lambda: current_app.extensions["security"]) - -_datastore = LocalProxy(lambda: _security.datastore) - -_pwd_context = LocalProxy(lambda: _security.pwd_context) - -_hashing_context = LocalProxy(lambda: _security.hashing_context) - -localize_callback = LocalProxy(lambda: _security.i18n_domain.gettext) - - - -## ... source file abbreviated to get to JSONEncoder examples ... - - - accept_mimetypes.best = best - if accept_mimetypes.best == "application/json": - return True - return False - - -def json_error_response(errors): - if isinstance(errors, str): - response_json = dict(error=errors) - elif isinstance(errors, dict): - response_json = dict(errors=errors) - else: - raise TypeError("The errors argument should be either a str or dict.") - - return response_json - -~~class FsJsonEncoder(JSONEncoder): - - def default(self, obj): - from .babel import is_lazy_string - - if is_lazy_string(obj): - return str(obj) - else: -~~ return JSONEncoder.default(self, obj) - - -class SmsSenderBaseClass(metaclass=abc.ABCMeta): - def __init__(self): - pass - - @abc.abstractmethod - def send_sms(self, from_number, to_number, msg): # pragma: no cover - return - - -class DummySmsSender(SmsSenderBaseClass): - def send_sms(self, from_number, to_number, msg): # pragma: no cover - return - - -class SmsSenderFactory: - senders = {"Dummy": DummySmsSender} - - @classmethod - def createSender(cls, name, *args, **kwargs): - return cls.senders[name](*args, **kwargs) - - - return _security._render_json(payload, code, headers=None, user=user) - - -def default_want_json(req): - if req.is_json: - return True - accept_mimetypes = req.accept_mimetypes - if not hasattr(req.accept_mimetypes, "best"): # pragma: no cover +from .babel import FsDomain +from .decorators import ( + default_reauthn_handler, + default_unauthn_handler, + default_unauthz_handler, +) +from .forms import ( + ChangePasswordForm, + ConfirmRegisterForm, + ForgotPasswordForm, + LoginForm, + PasswordlessLoginForm, + RegisterForm, + RegisterFormMixin, ## ... source file continues with no further JSONEncoder examples... diff --git a/content/pages/examples/flask/flask-json-jsonify.markdown b/content/pages/examples/flask/flask-json-jsonify.markdown index ae6a13eca..035721414 100644 --- a/content/pages/examples/flask/flask-json-jsonify.markdown +++ b/content/pages/examples/flask/flask-json-jsonify.markdown @@ -356,7 +356,7 @@ class UnbanUser(MethodView): data.append( { "id": user.id, - "type": "unban", + "type": "ban", "reverse": "ban", "reverse_name": _("Ban"), "reverse_url": url_for("management.ban_user", @@ -674,11 +674,15 @@ in Python scripts created by a developer. import os import uuid from importlib.util import module_from_spec, spec_from_file_location +from itertools import groupby +from operator import itemgetter import orjson from bs4 import BeautifulSoup +from bs4.element import Tag from bs4.formatter import HTMLFormatter -~~from flask import render_template, current_app, jsonify +~~from flask import current_app, jsonify, render_template +from jinja2.exceptions import TemplateNotFound def convert_to_snake_case(s): @@ -704,16 +708,15 @@ def get_component_module(module_name): user_specified_dir = current_app.config.get("MELD_COMPONENT_DIR", None) - ## ... source file abbreviated to get to jsonify examples ... - def render(self, component_name: str): - return self._view(component_name) - def _render_template(self, template_name: str, context_variables: dict): - return render_template(template_name, **context_variables) + try: + return render_template(template_name, **context_variables) + except TemplateNotFound: + return render_template(f"meld/{template_name}", **context_variables) def _view(self, component_name: str): data = self._attributes() @@ -724,7 +727,7 @@ def get_component_module(module_name): context_variables.update({"form": self._form}) rendered_template = self._render_template( - f"meld/{component_name}.html", context_variables + f"{component_name}.html", context_variables ) soup = BeautifulSoup(rendered_template, features="html.parser") @@ -755,9 +758,9 @@ def get_component_module(module_name): ) for model_attr in model_attrs: - element.attrs["value"] = context_variables[element.attrs[model_attr]] - - @staticmethod + value = context_variables[element.attrs[model_attr]] + element.attrs["value"] = value + if element.name == "select": ## ... source file continues with no further jsonify examples... @@ -770,7 +773,7 @@ def get_component_module(module_name): starter project to build a software-as-a-service (SaaS) web application in [Flask](/flask.html), with [Stripe](/stripe.html) for billing. The boilerplate relies on many common Flask extensions such as -[Flask-WTF](https://flask-wtf.readthedocs.io/en/latest/), +[Flask-WTF](https://flask-wtf.readthedocs.io/), [Flask-Login](https://flask-login.readthedocs.io/en/latest/), [Flask-Admin](https://flask-admin.readthedocs.io/en/latest/), and many others. The project is provided as open source under the @@ -1017,6 +1020,7 @@ import hashlib import sys from datetime import datetime +import sentry_sdk from authlib.oauth2 import OAuth2Error ~~from flask import flash, g, has_request_context, jsonify, render_template, request, session from itsdangerous import Signer @@ -1089,7 +1093,7 @@ def _format_request_data(data, hide_passwords=False): rv = {} for key, values in data: if hide_passwords and 'password' in key: - values = [v if not v else '<{} chars hidden>'.format(len(v)) for v in values] + values = [v if not v else f'<{len(v)} chars hidden>' for v in values] rv[key] = values if len(values) != 1 else values[0] return rv diff --git a/content/pages/examples/flask/flask-sessions-badsignature.markdown b/content/pages/examples/flask/flask-sessions-badsignature.markdown index c7b207921..afe01dee0 100644 --- a/content/pages/examples/flask/flask-sessions-badsignature.markdown +++ b/content/pages/examples/flask/flask-sessions-badsignature.markdown @@ -18,67 +18,7 @@ and SessionMixin are a couple of other callables within the `flask.sessions` package that also have code examples. -## Example 1 from FlaskBB -[FlaskBB](https://github.com/flaskbb/flaskbb) -([project website](https://flaskbb.org/)) is a [Flask](/flask.html)-based -forum web application. The web app allows users to chat in an open -message board or send private messages in plain text or -[Markdown](/markdown.html). - -FlaskBB is provided as open source -[under this license](https://github.com/flaskbb/flaskbb/blob/master/LICENSE). - -[**FlaskBB / flaskbb / tokens / serializer.py**](https://github.com/flaskbb/flaskbb/blob/master/flaskbb/tokens/serializer.py) - -```python -# serializer.py - -from datetime import timedelta - -~~from itsdangerous import (BadData, BadSignature, SignatureExpired, - TimedJSONWebSignatureSerializer) - -from ..core import tokens - - -_DEFAULT_EXPIRY = timedelta(hours=1) - - -class FlaskBBTokenSerializer(tokens.TokenSerializer): - - def __init__(self, secret_key, expiry=_DEFAULT_EXPIRY): - self._serializer = TimedJSONWebSignatureSerializer( - secret_key, int(expiry.total_seconds()) - ) - - def dumps(self, token): - return self._serializer.dumps( - { - 'id': token.user_id, - 'op': token.operation, - } - ) - - def loads(self, raw_token): - try: - parsed = self._serializer.loads(raw_token) - except SignatureExpired: - raise tokens.TokenError.expired() -~~ except BadSignature: # pragma: no branch - raise tokens.TokenError.invalid() - except BadData: # pragma: no cover - raise tokens.TokenError.bad() - else: - return tokens.Token(user_id=parsed['id'], operation=parsed['op']) - - - -## ... source file continues with no further BadSignature examples... - -``` - - -## Example 2 from flask-base +## Example 1 from flask-base [flask-base](https://github.com/hack4impact/flask-base) ([project documentation](http://hack4impact.github.io/flask-base/)) provides boilerplate code for new [Flask](/flask.html) web apps. @@ -88,7 +28,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) backend, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling and many others. flask-base is provided as open source under the @@ -219,7 +159,7 @@ class Role(db.Model): ``` -## Example 3 from flask-bones +## Example 2 from flask-bones [flask-bones](https://github.com/cburmeister/flask-bones) ([demo](http://flask-bones.herokuapp.com/)) is large scale [Flask](/flask.html) example application built @@ -319,7 +259,7 @@ def verify(token): ``` -## Example 4 from Flask-Security-Too +## Example 3 from Flask-Security-Too [Flask-Security-Too](https://github.com/Flask-Middleware/flask-security/) ([PyPi page](https://pypi.org/project/Flask-Security-Too/) and [project documentation](https://flask-security-too.readthedocs.io/en/stable/)) @@ -338,9 +278,8 @@ The Flask-Security-Too project is provided as open source under the ```python # utils.py -from typing import Dict, List +import typing as t import warnings -from datetime import timedelta from urllib.parse import parse_qsl, parse_qs, urlsplit, urlunsplit, urlencode import urllib.request import urllib.error @@ -352,6 +291,7 @@ from flask import ( flash, g, request, + render_template, session, url_for, ) @@ -364,19 +304,20 @@ from flask_principal import AnonymousIdentity, Identity, identity_changed, Need from flask_wtf import csrf from wtforms import ValidationError ~~from itsdangerous import BadSignature, SignatureExpired +from werkzeug import __version__ as werkzeug_version from werkzeug.local import LocalProxy from werkzeug.datastructures import MultiDict from .quart_compat import best, get_quart_status +from .proxies import _security, _datastore, _pwd_context, _hashing_context from .signals import user_authenticated -_security = LocalProxy(lambda: current_app.extensions["security"]) +if t.TYPE_CHECKING: # pragma: no cover + from flask import Flask, Response + from .datastore import User -_datastore = LocalProxy(lambda: _security.datastore) +SB = t.Union[str, bytes] -_pwd_context = LocalProxy(lambda: _security.pwd_context) - -_hashing_context = LocalProxy(lambda: _security.hashing_context) localize_callback = LocalProxy(lambda: _security.i18n_domain.gettext) @@ -389,21 +330,20 @@ def _(translate): - ## ... source file abbreviated to get to BadSignature examples ... - if config_value("EMAIL_HTML"): - html = _security.render_template("%s/%s.html" % ctx, **context) - - subject = localize_callback(subject) - sender = _security.email_sender if isinstance(sender, LocalProxy): sender = sender._get_current_object() + if isinstance(sender, tuple) and len(sender) == 2: + sender = (str(sender[0]), str(sender[1])) + else: + sender = str(sender) + _security._mail_util.send_mail( - template, subject, recipient, str(sender), body, html, context.get("user", None) + template, subject, recipient, sender, body, html, context.get("user", None) ) @@ -432,8 +372,10 @@ def get_token_status(token, serializer, max_age=None, return_data=False): return expired, invalid, user -def check_and_get_token_status(token, serializer, within=None): - serializer = getattr(_security, serializer + "_serializer") +def check_and_get_token_status( + token: str, serializer_name: str, within: datetime.timedelta +) -> t.Tuple[bool, bool, t.Any]: + serializer = getattr(_security, serializer_name + "_serializer") max_age = within.total_seconds() data = None expired, invalid = False, False @@ -449,7 +391,7 @@ def check_and_get_token_status(token, serializer, within=None): return expired, invalid, data -def get_identity_attributes(app=None) -> List: +def get_identity_attributes(app: t.Optional["Flask"] = None) -> t.List[str]: app = app or current_app iattrs = app.config["SECURITY_USER_IDENTITY_ATTRIBUTES"] if iattrs: @@ -457,7 +399,9 @@ def get_identity_attributes(app=None) -> List: return [] -def get_identity_attribute(attr, app=None) -> Dict: +def get_identity_attribute( + attr: str, app: t.Optional["Flask"] = None +) -> t.Dict[str, t.Any]: app = app or current_app iattrs = app.config["SECURITY_USER_IDENTITY_ATTRIBUTES"] if iattrs: @@ -466,8 +410,6 @@ def get_identity_attribute(attr, app=None) -> Dict: ] if details: return details[0] - return {} - ## ... source file continues with no further BadSignature examples... diff --git a/content/pages/examples/flask/flask-signals-got-request-exception.markdown b/content/pages/examples/flask/flask-signals-got-request-exception.markdown index dcce6fb02..dc2a466c4 100644 --- a/content/pages/examples/flask/flask-signals-got-request-exception.markdown +++ b/content/pages/examples/flask/flask-signals-got-request-exception.markdown @@ -49,7 +49,10 @@ from types import MethodType from flask import url_for, request, current_app from flask import make_response as original_flask_make_response -from flask.helpers import _endpoint_from_view_func +try: + from flask.helpers import _endpoint_from_view_func +except ImportError: + from flask.scaffold import _endpoint_from_view_func ~~from flask.signals import got_request_exception from jsonschema import RefResolver @@ -63,54 +66,25 @@ from werkzeug.exceptions import ( NotAcceptable, InternalServerError, ) -from werkzeug.wrappers import BaseResponse + +from werkzeug import __version__ as werkzeug_version + +if werkzeug_version.split('.')[0] >= '2': + from werkzeug.wrappers import Response as BaseResponse +else: + from werkzeug.wrappers import BaseResponse from . import apidoc from .mask import ParseError, MaskError from .namespace import Namespace from .postman import PostmanCollectionV1 -from .resource import Resource -from .swagger import Swagger -from .utils import default_id, camel_to_dash, unpack -from .representations import output_json -from ._http import HTTPStatus - ## ... source file abbreviated to get to got_request_exception examples ... - except MethodNotAllowed as e: - valid_route_method = e.valid_methods[0] - rule, _ = adapter.match(method=valid_route_method, return_rule=True) - return self.owns_endpoint(rule.endpoint) - except NotFound: - return self.catch_all_404s - except Exception: - pass - - def _has_fr_route(self): - if self._should_use_fr_error_handler(): - return True - if not request.url_rule: - return False - return self.owns_endpoint(request.url_rule.endpoint) - - def error_router(self, original_handler, e): - if self._has_fr_route(): - try: - return self.handle_error(e) - except Exception as f: - return original_handler(f) - return original_handler(e) - - def handle_error(self, e): -~~ got_request_exception.send(current_app._get_current_object(), exception=e) - - if ( - not isinstance(e, HTTPException) and current_app.propagate_exceptions - and not isinstance(e, tuple(self.error_handlers.keys())) + and not isinstance(e, tuple(self._own_and_child_error_handlers.keys())) ): exc_type, exc_value, tb = sys.exc_info() @@ -130,6 +104,35 @@ from ._http import HTTPStatus if isinstance(e, typecheck): result = handler(e) default_data, code, headers = unpack( + result, HTTPStatus.INTERNAL_SERVER_ERROR + ) + break + else: +~~ got_request_exception.send(current_app._get_current_object(), exception=e) + + if isinstance(e, HTTPException): + code = HTTPStatus(e.code) + if include_message_in_response: + default_data = {"message": getattr(e, "description", code.phrase)} + headers = e.get_response().headers + elif self._default_error_handler: + result = self._default_error_handler(e) + default_data, code, headers = unpack( + result, HTTPStatus.INTERNAL_SERVER_ERROR + ) + else: + code = HTTPStatus.INTERNAL_SERVER_ERROR + if include_message_in_response: + default_data = { + "message": code.phrase, + } + + if include_message_in_response: + default_data["message"] = default_data.get("message", str(e)) + + data = getattr(e, "data", default_data) + fallback_mediatype = None + ## ... source file continues with no further got_request_exception examples... diff --git a/content/pages/examples/flask/flask-templating-render-template-string.markdown b/content/pages/examples/flask/flask-templating-render-template-string.markdown index 563672ef3..7f74cb079 100644 --- a/content/pages/examples/flask/flask-templating-render-template-string.markdown +++ b/content/pages/examples/flask/flask-templating-render-template-string.markdown @@ -20,7 +20,7 @@ the `.templating` part. render_template is another callable from the `flask.templating` package with code examples. -These topics are also useful while reading the `render_template_string` examples: +These subjects go along with the `render_template_string` code examples: * [template engines](/template-engines.html), specifically [Jinja2](/jinja2.html) * [Flask](/flask.html) and the concepts for [web frameworks](/web-frameworks.html) @@ -75,26 +75,26 @@ def test_themes_cant_access_configpy_attributes(): ## ... source file abbreviated to get to render_template_string examples ... - r = client.get("/themes/foo/static/js/pages/main.dev.js") + except TemplateNotFound: + pass + try: + r = client.get("/themes/foo_fallback/static/js/pages/main.dev.js") except TemplateNotFound: pass destroy_ctfd(app) - class ThemeFallbackConfig(TestingConfig): - THEME_FALLBACK = True - - app = create_ctfd(config=ThemeFallbackConfig) + app = create_ctfd() with app.app_context(): - set_config("ctf_theme", "foo") + set_config("ctf_theme", "foo_fallback") assert app.config["THEME_FALLBACK"] == True with app.test_client() as client: r = client.get("/") assert r.status_code == 200 - r = client.get("/themes/foo/static/js/pages/main.dev.js") + r = client.get("/themes/foo_fallback/static/js/pages/main.dev.js") assert r.status_code == 200 destroy_ctfd(app) - os.rmdir(os.path.join(app.root_path, "themes", "foo")) + os.rmdir(os.path.join(app.root_path, "themes", "foo_fallback")) def test_theme_template_loading_by_prefix(): @@ -110,9 +110,10 @@ def test_theme_template_disallow_loading_admin_templates(): with app.app_context(): try: filename = os.path.join( - app.root_path, "themes", "foo", "admin", "malicious.html" + app.root_path, "themes", "foo_disallow", "admin", "malicious.html" ) os.makedirs(os.path.dirname(filename), exist_ok=True) + set_config("ctf_theme", "foo_disallow") with open(filename, "w") as f: f.write("malicious") @@ -120,7 +121,8 @@ def test_theme_template_disallow_loading_admin_templates(): ~~ render_template_string("{% include 'admin/malicious.html' %}") finally: shutil.rmtree( - os.path.join(app.root_path, "themes", "foo"), ignore_errors=True + os.path.join(app.root_path, "themes", "foo_disallow"), + ignore_errors=True, ) diff --git a/content/pages/examples/flask/flask-templating-render-template.markdown b/content/pages/examples/flask/flask-templating-render-template.markdown index f47d4a348..9d320fab4 100644 --- a/content/pages/examples/flask/flask-templating-render-template.markdown +++ b/content/pages/examples/flask/flask-templating-render-template.markdown @@ -21,7 +21,7 @@ the `.templating` part. render_template_string is another callable from the `flask.templating` package with code examples. -You should read up on these subjects along with these `render_template` examples: +These topics are also useful while reading the `render_template` examples: * [template engines](/template-engines.html), specifically [Jinja2](/jinja2.html) * [Flask](/flask.html) and the concepts for [web frameworks](/web-frameworks.html) @@ -167,25 +167,25 @@ def test_themes_cant_access_configpy_attributes(): ## ... source file abbreviated to get to render_template examples ... + pass + try: + r = client.get("/themes/foo_fallback/static/js/pages/main.dev.js") except TemplateNotFound: pass destroy_ctfd(app) - class ThemeFallbackConfig(TestingConfig): - THEME_FALLBACK = True - - app = create_ctfd(config=ThemeFallbackConfig) + app = create_ctfd() with app.app_context(): - set_config("ctf_theme", "foo") + set_config("ctf_theme", "foo_fallback") assert app.config["THEME_FALLBACK"] == True with app.test_client() as client: r = client.get("/") assert r.status_code == 200 - r = client.get("/themes/foo/static/js/pages/main.dev.js") + r = client.get("/themes/foo_fallback/static/js/pages/main.dev.js") assert r.status_code == 200 destroy_ctfd(app) - os.rmdir(os.path.join(app.root_path, "themes", "foo")) + os.rmdir(os.path.join(app.root_path, "themes", "foo_fallback")) def test_theme_template_loading_by_prefix(): @@ -201,9 +201,10 @@ def test_theme_template_disallow_loading_admin_templates(): with app.app_context(): try: filename = os.path.join( - app.root_path, "themes", "foo", "admin", "malicious.html" + app.root_path, "themes", "foo_disallow", "admin", "malicious.html" ) os.makedirs(os.path.dirname(filename), exist_ok=True) + set_config("ctf_theme", "foo_disallow") with open(filename, "w") as f: f.write("malicious") @@ -211,7 +212,8 @@ def test_theme_template_disallow_loading_admin_templates(): render_template_string("{% include 'admin/malicious.html' %}") finally: shutil.rmtree( - os.path.join(app.root_path, "themes", "foo"), ignore_errors=True + os.path.join(app.root_path, "themes", "foo_disallow"), + ignore_errors=True, ) @@ -310,7 +312,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) backend, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling and many others. flask-base is provided as open source under the @@ -801,7 +803,7 @@ def update(c_id): is an example application that ties together the [intTellInput.js](https://github.com/jackocnr/intl-tel-input) JavaScript plugin with the -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) form-handling +[Flask-WTF](https://flask-wtf.readthedocs.io/) form-handling library. flask-phone-input is provided as open source under the [MIT license](https://github.com/miguelgrinberg/flask-phone-input/blob/1a1c227c044474ce0fe133493d7f8b0fb8312409/LICENSE). @@ -869,11 +871,15 @@ in Python scripts created by a developer. import os import uuid from importlib.util import module_from_spec, spec_from_file_location +from itertools import groupby +from operator import itemgetter import orjson from bs4 import BeautifulSoup +from bs4.element import Tag from bs4.formatter import HTMLFormatter -~~from flask import render_template, current_app, jsonify +~~from flask import current_app, jsonify, render_template +from jinja2.exceptions import TemplateNotFound def convert_to_snake_case(s): @@ -899,11 +905,9 @@ def get_component_module(module_name): user_specified_dir = current_app.config.get("MELD_COMPONENT_DIR", None) - ## ... source file abbreviated to get to render_template examples ... - func for func in dir(self) if callable(getattr(self, func)) and not func.startswith("_") @@ -928,7 +932,10 @@ def get_component_module(module_name): return self._view(component_name) def _render_template(self, template_name: str, context_variables: dict): -~~ return render_template(template_name, **context_variables) + try: +~~ return render_template(template_name, **context_variables) + except TemplateNotFound: +~~ return render_template(f"meld/{template_name}", **context_variables) def _view(self, component_name: str): data = self._attributes() @@ -939,7 +946,7 @@ def get_component_module(module_name): context_variables.update({"form": self._form}) rendered_template = self._render_template( - f"meld/{component_name}.html", context_variables + f"{component_name}.html", context_variables ) soup = BeautifulSoup(rendered_template, features="html.parser") @@ -1020,7 +1027,7 @@ def ui_for(api): starter project to build a software-as-a-service (SaaS) web application in [Flask](/flask.html), with [Stripe](/stripe.html) for billing. The boilerplate relies on many common Flask extensions such as -[Flask-WTF](https://flask-wtf.readthedocs.io/en/latest/), +[Flask-WTF](https://flask-wtf.readthedocs.io/), [Flask-Login](https://flask-login.readthedocs.io/en/latest/), [Flask-Admin](https://flask-admin.readthedocs.io/en/latest/), and many others. The project is provided as open source under the @@ -1218,114 +1225,7 @@ def paySuccess(): ``` -## Example 13 from Flask-Security-Too -[Flask-Security-Too](https://github.com/Flask-Middleware/flask-security/) -([PyPi page](https://pypi.org/project/Flask-Security-Too/) and -[project documentation](https://flask-security-too.readthedocs.io/en/stable/)) -is a maintained fork of the original -[Flask-Security](https://github.com/mattupstate/flask-security) project that -makes it easier to add common security features to [Flask](/flask.html) -web applications. A few of the critical goals of the Flask-Security-Too -project are ensuring JavaScript client-based single-page applications (SPAs) -can work securely with Flask-based backends and that guidance by the -[OWASP](https://owasp.org/) organization is followed by default. - -The Flask-Security-Too project is provided as open source under the -[MIT license](https://github.com/Flask-Middleware/flask-security/blob/master/LICENSE). - -[**Flask-Security-Too / flask_security / core.py**](https://github.com/Flask-Middleware/flask-security/blob/master/flask_security/./core.py) - -```python -# core.py - -from datetime import datetime, timedelta -import warnings - -import pkg_resources -~~from flask import _request_ctx_stack, current_app, render_template -from flask_login import AnonymousUserMixin, LoginManager -from flask_login import UserMixin as BaseUserMixin -from flask_login import current_user -from flask_principal import Identity, Principal, RoleNeed, UserNeed, identity_loaded -from itsdangerous import URLSafeTimedSerializer -from passlib.context import CryptContext -from werkzeug.datastructures import ImmutableList -from werkzeug.local import LocalProxy - -from .babel import get_i18n_domain, have_babel -from .decorators import ( - default_reauthn_handler, - default_unauthn_handler, - default_unauthz_handler, -) -from .forms import ( - ChangePasswordForm, - ConfirmRegisterForm, - ForgotPasswordForm, - LoginForm, - PasswordlessLoginForm, - RegisterForm, - ResetPasswordForm, - SendConfirmationForm, - - -## ... source file abbreviated to get to render_template examples ... - - - sms_service = cv("SMS_SERVICE", app=app) - if sms_service == "Twilio": # pragma: no cover - self._check_modules("twilio", "SMS") - if state.phone_util_cls == PhoneUtil: - self._check_modules("phonenumbers", "SMS") - - secrets = cv("TOTP_SECRETS", app=app) - issuer = cv("TOTP_ISSUER", app=app) - if not secrets or not issuer: - raise ValueError("Both TOTP_SECRETS and TOTP_ISSUER must be set") - state.totp_factory(state.totp_cls(secrets, issuer)) - - if cv("PASSWORD_COMPLEXITY_CHECKER", app=app) == "zxcvbn": - self._check_modules("zxcvbn", "PASSWORD_COMPLEXITY_CHECKER") - return state - - def _check_modules(self, module, config_name): # pragma: no cover - from importlib.util import find_spec - - module_exists = find_spec(module) - if not module_exists: - raise ValueError(f"{module} is required for {config_name}") - - return module_exists - -~~ def render_template(self, *args, **kwargs): -~~ return render_template(*args, **kwargs) - - def render_json(self, cb): - self._state._render_json = cb - - def want_json(self, fn): - self._state._want_json = fn - - def unauthz_handler(self, cb): - self._state._unauthz_handler = cb - - def unauthn_handler(self, cb): - self._state._unauthn_handler = cb - - def reauthn_handler(self, cb): - self._state._reauthn_handler = cb - - def __getattr__(self, name): - return getattr(self._state, name, None) - - - -## ... source file continues with no further render_template examples... - -``` - - -## Example 14 from Flask-SocketIO +## Example 13 from Flask-SocketIO [Flask-SocketIO](https://github.com/miguelgrinberg/Flask-SocketIO) ([PyPI package information](https://pypi.org/project/Flask-SocketIO/), [official tutorial](https://blog.miguelgrinberg.com/post/easy-websockets-with-flask-and-gevent) @@ -1401,7 +1301,7 @@ def get_session(): ``` -## Example 15 from Flask-User +## Example 14 from Flask-User [Flask-User](https://github.com/lingthio/Flask-User) ([PyPI information](https://pypi.org/project/Flask-User/) and @@ -1493,7 +1393,7 @@ class EmailManager(object): ``` -## Example 16 from Flasky +## Example 15 from Flasky [Flasky](https://github.com/miguelgrinberg/flasky) is a wonderful example application by [Miguel Grinberg](https://github.com/miguelgrinberg) that he builds @@ -1533,7 +1433,7 @@ def send_email(to, subject, template, **kwargs): ``` -## Example 17 from Datadog Flask Example App +## Example 16 from Datadog Flask Example App The [Datadog Flask example app](https://github.com/DataDog/trace-examples/tree/master/python/flask) contains many examples of the [Flask](/flask.html) core functions available to a developer using the [web framework](/web-frameworks.html). @@ -1635,7 +1535,7 @@ app.url_map.add(Rule('/custom-endpoint/', endpoint='custom-endpoint')) ``` -## Example 18 from indico +## Example 17 from indico [indico](https://github.com/indico/indico) ([project website](https://getindico.io/), [documentation](https://docs.getindico.io/en/stable/installation/) @@ -1663,7 +1563,7 @@ class MathjaxMixin: ``` -## Example 19 from keras-flask-deploy-webapp +## Example 18 from keras-flask-deploy-webapp The [keras-flask-deploy-webapp](https://github.com/mtobeiyf/keras-flask-deploy-webapp) project combines the [Flask](/flask.html) [web framework](/web-frameworks.html) @@ -1755,7 +1655,7 @@ if __name__ == '__main__': ``` -## Example 20 from newspie +## Example 19 from newspie [NewsPie](https://github.com/skamieniarz/newspie) is a minimalistic news aggregator created with [Flask](/flask.html) and the [News API](https://newsapi.org/). @@ -1771,6 +1671,7 @@ import configparser import json import logging import os +from typing import Union import requests import requests_cache @@ -1796,15 +1697,16 @@ requests_cache.install_cache(cache_name='news_cache', expire_after=300) APP = Flask(__name__) +SESSION = requests.Session() +SESSION.headers.update({'Authorization': API_KEY}) -@APP.route('/', methods=['GET', 'POST']) -def root(): ## ... source file abbreviated to get to render_template examples ... + return redirect(url_for('category', category='general', page=1)) @APP.route('/category/', methods=['GET', 'POST']) @@ -1819,14 +1721,13 @@ def category(category): country = get_cookie('country') if country is not None: params.update({'country': country}) - response = requests.get(TOP_HEADLINES, - params=params, - headers={'Authorization': API_KEY}) + response = SESSION.get(TOP_HEADLINES, params=params) if response.status_code == 200: pages = count_pages(response.json()) if page > pages: page = pages - return redirect(url_for('category', category=category, page=page)) + return redirect( + url_for('category', category=category, page=page)) articles = parse_articles(response.json()) return render(articles, page, pages, country, category) elif response.status_code == 401: @@ -1835,7 +1736,7 @@ def category(category): @APP.route('/search/', methods=['GET', 'POST']) -def search(query): +def search(query: str): page = request.args.get('page', default=1, type=int) if page < 1: return redirect(url_for('search', query=query, page=1)) @@ -1847,19 +1748,20 @@ def search(query): } if request.method == 'POST': return do_post(page, category='search', current_query=query) - response = requests.get(EVERYTHING, - params=params, - headers={'Authorization': API_KEY}) + response = SESSION.get(EVERYTHING, params=params) pages = count_pages(response.json()) if page > pages: page = pages return redirect(url_for('search', query=query, page=page)) articles = parse_articles(response.json()) + return render(articles, + page, ## ... source file abbreviated to get to render_template examples ... + parsed_articles = [] if response.get('status') == 'ok': for article in response.get('articles'): parsed_articles.append({ @@ -1876,11 +1778,10 @@ def search(query): return parsed_articles -def count_pages(response): - pages = 0 +def count_pages(response: dict) -> int: if response.get('status') == 'ok': - pages = (-(-response.get('totalResults', 0) // PAGE_SIZE)) - return pages + return (-(-response.get('totalResults', 0) // PAGE_SIZE)) + return 0 def render(articles, page, pages, country, category): @@ -1895,9 +1796,8 @@ def render(articles, page, pages, country, category): pages=pages) -def get_cookie(key): - cookie_value = request.cookies.get(key) - return cookie_value +def get_cookie(key: str) -> Union[str, None]: + return request.cookies.get(key) if __name__ == '__main__': @@ -1910,7 +1810,7 @@ if __name__ == '__main__': ``` -## Example 21 from Science Flask +## Example 20 from Science Flask [Science Flask](https://github.com/danielhomola/science_flask) is a [Flask](/flask.html)-powered web application for online scientific research tools. The project was built as a template @@ -2198,7 +2098,7 @@ def static_from_root(): ``` -## Example 22 from tedivms-flask +## Example 21 from tedivms-flask [tedivm's flask starter app](https://github.com/tedivm/tedivms-flask) is a base of [Flask](/flask.html) code and related projects such as [Celery](/celery.html) which provides a template to start your own @@ -2301,7 +2201,7 @@ def init_error_handlers(app): ``` -## Example 23 from trape +## Example 22 from trape [trape](https://github.com/jofpin/trape) is a research tool for tracking people's activities that are logged digitally. The tool uses [Flask](/flask.html) to create a web front end to view aggregated data @@ -2319,6 +2219,7 @@ import os ~~from flask import Flask, render_template, session, request, json, redirect, url_for, send_from_directory from flask_cors import CORS from trape import Trape +import urllib from core.db import Database trape = Trape(1) @@ -2342,7 +2243,6 @@ def index(): return trape.injectCSS_Paths(render_template("/login.html").replace('[LOGIN_SRC]', trape.JSFiles[2]['src']).replace('[LIBS_SRC]', trape.JSFiles[1]['src'])) - ## ... source file abbreviated to get to render_template examples ... diff --git a/content/pages/examples/flask/flask-views-methodview.markdown b/content/pages/examples/flask/flask-views-methodview.markdown index 00e7805b0..c0e34c8e9 100644 --- a/content/pages/examples/flask/flask-views-methodview.markdown +++ b/content/pages/examples/flask/flask-views-methodview.markdown @@ -19,7 +19,7 @@ and http_method_funcs are a couple of other callables within the `flask.views` package that also have code examples. -These topics are also useful while reading the `MethodView` examples: +You should read up on these subjects along with these `MethodView` examples: * [web development](/web-development.html) and [web design](/web-design.html) * [web framework concepts](/web-frameworks.html) and the [Flask framework](/flask.html) @@ -430,7 +430,12 @@ from __future__ import unicode_literals from flask import request ~~from flask.views import MethodView -from werkzeug.wrappers import BaseResponse +from werkzeug import __version__ as werkzeug_version + +if werkzeug_version.split('.')[0] >= '2': + from werkzeug.wrappers import Response as BaseResponse +else: + from werkzeug.wrappers import BaseResponse from .model import ModelBase diff --git a/content/pages/examples/flask/flask-views-view.markdown b/content/pages/examples/flask/flask-views-view.markdown index 25ca74e2d..1f8e51936 100644 --- a/content/pages/examples/flask/flask-views-view.markdown +++ b/content/pages/examples/flask/flask-views-view.markdown @@ -20,7 +20,7 @@ and http_method_funcs are a couple of other callables within the `flask.views` package that also have code examples. -These topics are also useful while reading the `View` examples: +You should read up on these subjects along with these `View` examples: * [web development](/web-development.html) and [web design](/web-design.html) * [web framework concepts](/web-frameworks.html) and the [Flask framework](/flask.html) diff --git a/content/pages/examples/sqlalchemy/sqlalchemy-exc-integrityerror.markdown b/content/pages/examples/sqlalchemy/sqlalchemy-exc-integrityerror.markdown index 615febd9c..88658c4f4 100644 --- a/content/pages/examples/sqlalchemy/sqlalchemy-exc-integrityerror.markdown +++ b/content/pages/examples/sqlalchemy/sqlalchemy-exc-integrityerror.markdown @@ -142,7 +142,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) backend, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling and many others. flask-base is provided as open source under the diff --git a/content/pages/examples/sqlalchemy/sqlalchemy-extensions-plug-ins.markdown b/content/pages/examples/sqlalchemy/sqlalchemy-extensions-plug-ins.markdown index 0ff095a54..c24866e5f 100644 --- a/content/pages/examples/sqlalchemy/sqlalchemy-extensions-plug-ins.markdown +++ b/content/pages/examples/sqlalchemy/sqlalchemy-extensions-plug-ins.markdown @@ -49,7 +49,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) back end, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling, and many others. flask-base is provided as open source under the diff --git a/content/pages/meta/00-change-log.markdown b/content/pages/meta/00-change-log.markdown index 9f52bf150..bf6137a13 100644 --- a/content/pages/meta/00-change-log.markdown +++ b/content/pages/meta/00-change-log.markdown @@ -13,7 +13,38 @@ view commit-level changes via the [source repository's commit log](https://github.com/mattmakai/fullstackpython.com/commits/) on GitHub. +## 2022 +### October +* Starting to get back into updating this site again! Note that I'll probably + spend most of my side project time on [Plushcap](https://www.plushcap.com/) + but I'm removing old resources and adding new good ones on here to keep the + site maintained. + +### March +* I decided to go so minimal that I removed the logo to make the site load + faster, and also got rid of some unncessary CSS on front page. + +### February +* Still on break, but updated the header and footer. Also made some website + tweaks to make it faster. + ## 2021 +### October +* Taking a break for a few months to recharge and work on a different coding + side project that I'm excited about. Be back in a few months with some updates. + +### September +* Clean up continues across the site to remove link rot and replace + out-of-date links with newer ones where necessary. + +### August +* New blog post on + [Application Performance Monitoring AWS Lambda Functions with Sentry](/blog/application-performance-monitoring-aws-lambda-functions-sentry.html). + +### June +* Fix broken links across the site. It's a real shame people and companies don't use + 301 redirects more often rather than just breaking their URL structures. + ### May * New resources on [GPT-3](/gpt-3.html), [Jupyter Notebook](/jupyter-notebook.html) and [serverless](/serverless.html) pages. diff --git a/content/posts/160518-install-postgresql-python-3-ubuntu-1604.markdown b/content/posts/160518-install-postgresql-python-3-ubuntu-1604.markdown index e5d4338cc..baf38465d 100644 --- a/content/posts/160518-install-postgresql-python-3-ubuntu-1604.markdown +++ b/content/posts/160518-install-postgresql-python-3-ubuntu-1604.markdown @@ -12,7 +12,7 @@ headeralt: PostgreSQL and Ubuntu logos. Copyright their respective owners. [PostgreSQL](/postgresql.html) is a powerful open source [relational database](/databases.html) frequently used to create, read, update and delete [Python web application](/web-frameworks.html) data. -[Psycopg2](http://initd.org/psycopg/) is a PostgreSQL database +[Psycopg2](https://www.psycopg.org/) is a PostgreSQL database driver that serves as a Python client for access to the PostgreSQL server. This post explains how to install PostgreSQL on [Ubuntu 16.04](/ubuntu.html) and run a few basic SQL queries within a Python program. @@ -110,7 +110,7 @@ found in the ## Installing psycopg2 Now that PostgreSQL is installed and we have a non-superuser account, we -can install the [psycopg2](http://initd.org/psycopg/) package. Let's +can install the [psycopg2](https://www.psycopg.org/) package. Let's figure out where our `python3` executable is located, create a virtualenv with `python3`, activate the virtualenv and then install the psycopg2 package with `pip`. Find your `python3` executable using the `which` command. diff --git a/content/posts/160626-django-gunicorn-mint-linux-17.markdown b/content/posts/160626-django-gunicorn-mint-linux-17.markdown index 688adaaf8..8ab8836ff 100644 --- a/content/posts/160626-django-gunicorn-mint-linux-17.markdown +++ b/content/posts/160626-django-gunicorn-mint-linux-17.markdown @@ -3,7 +3,7 @@ slug: python-3-django-gunicorn-linux-mint-17 meta: A step-by-step walkthrough on configuring Linux Mint 17.3 with Python 3, Django and Green Unicorn (Gunicorn). category: post date: 2016-06-26 -modified: 2016-07-22 +modified: 2021-06-17 newsletter: False headerimage: /img/160626-mint-django-gunicorn/header.jpg headeralt: Django, Green Unicorn and Linux Mint logos. Copyright their respective owners. @@ -32,7 +32,7 @@ their current versions as of June 2016 are: * [Python](/why-use-python.html) version [3.5.1](https://www.python.org/downloads/release/python-351/) * [Django](/django.html) web framework version - [1.9.7](https://docs.djangoproject.com/en/1.9/releases/1.9/) + [1.9.x](https://pypi.org/project/Django/1.9.13/) * [Green Unicorn (Gunicorn)](/green-unicorn-gunicorn.html) version [19.6](http://docs.gunicorn.org/en/stable/news.html) diff --git a/content/posts/170723-monitor-flask-apps.markdown b/content/posts/170723-monitor-flask-apps.markdown index b3cd8b760..4ef02bfc6 100644 --- a/content/posts/170723-monitor-flask-apps.markdown +++ b/content/posts/170723-monitor-flask-apps.markdown @@ -165,7 +165,7 @@ The above [Jinja2](/jinja2.html) template is basic HTML without any [embedded template tags](http://jinja.pocoo.org/docs/latest/templates/). The template creates a very plain page with a header description of "PUBG so good" and a GIF from this -[excellent computer game](http://store.steampowered.com/app/578080/PLAYERUNKNOWNS_BATTLEGROUNDS/). +[excellent computer game](https://store.steampowered.com/app/578080/PUBG_BATTLEGROUNDS/). Time to run and test our code. Change into the base directory of your project where `app.py` file is located. Execute `app.py` using the `python` diff --git a/content/posts/190626-dev-led-sales-startups.markdown b/content/posts/190626-dev-led-sales-startups.markdown index 55e4ec431..ba8066e9e 100644 --- a/content/posts/190626-dev-led-sales-startups.markdown +++ b/content/posts/190626-dev-led-sales-startups.markdown @@ -456,9 +456,9 @@ tactics.

Conferences split into a couple of categories: community-run and vendor-run. For example, -WWDC is Apple's vendor-run +WWDC is Apple's vendor-run developer conference. They run the show and control the messaging. -PyCon US is the community-run +PyCon US is the community-run Python developer conference. There are a ton of vendors there as sponsors but no one company controls what happens at the conference.

diff --git a/content/posts/200308-financial-resources-developers.markdown b/content/posts/200308-financial-resources-developers.markdown index fb0619334..80f2560d0 100644 --- a/content/posts/200308-financial-resources-developers.markdown +++ b/content/posts/200308-financial-resources-developers.markdown @@ -77,7 +77,7 @@ macroeconomic data trends. is well-written, straightforward and accessible, particularly because it clearly targets its software developer audience. -* [Don't Quit Your Day Job](https://dqydj.com/) uses a ton of metrics +* [Don't Quit Your Day Job](https://dqydj.com) uses a ton of metrics and statistics to ground their articles on financial topics that are often relevant specifically to software developers. For example, the article on diff --git a/content/posts/200630-report-errors-flask-web-apps-sentry.markdown b/content/posts/200630-report-errors-flask-web-apps-sentry.markdown index e55c14c5c..2e9ec9684 100644 --- a/content/posts/200630-report-errors-flask-web-apps-sentry.markdown +++ b/content/posts/200630-report-errors-flask-web-apps-sentry.markdown @@ -260,5 +260,5 @@ or [@mattmakai](https://twitter.com/mattmakai). I am also on GitHub with the username [mattmakai](https://github.com/mattmakai). If you see an issue or error in this tutorial, please -[fork the source repository on GitHub](https://github.com/mattmakai/fullstackpython/blob/master/content/posts/200630-report-errors-flask-web-apps-sentry.markdown) +[fork the source repository on GitHub](https://github.com/mattmakai/fullstackpython.com/blob/master/content/posts/200630-report-errors-flask-web-apps-sentry.markdown) and submit a pull request with the fix. diff --git a/content/posts/200809-transcribe-recordings-speech-text-assemblyai.markdown b/content/posts/200809-transcribe-recordings-speech-text-assemblyai.markdown index 96926ddc2..4a5027fbe 100644 --- a/content/posts/200809-transcribe-recordings-speech-text-assemblyai.markdown +++ b/content/posts/200809-transcribe-recordings-speech-text-assemblyai.markdown @@ -3,7 +3,7 @@ slug: transcribe-recordings-speech-text-assemblyai meta: Learn to transcribe speech in recordings like MP3s into text with Python and AssemblyAI's API. category: post date: 2020-08-09 -modified: 2021-01-18 +modified: 2021-09-13 newsletter: False headerimage: /img/headers/python-assemblyai.jpg headeralt: Logos for the implementations used in this blog post. Copyright their respective owners. @@ -457,9 +457,7 @@ That's it, we've got our transcription! You may be wondering what to do if the accuracy isn't where you need it to be for your situation. That is where [boosting accuracy for keywords or phrases](https://docs.assemblyai.com/guides/boosting-accuracy-for-keywords-or-phrases) -and -[selecting a model that better matches your data](https://docs.assemblyai.com/guides/transcribing-with-a-different-acoustic-or-custom-language-model) -come in. You can use either of those two methods to boost the accuracy +comes in. You can use either of those two methods to boost the accuracy of your recordings to an acceptable level for your situation. @@ -470,7 +468,7 @@ transcribe recordings with speech into text output. Next, take a look at some of their more advanced documentation that goes beyond the basics in this tutorial: -* [Supported file formats](https://docs.assemblyai.com/overview/supported-file-formats) +* [Supported file formats](https://docs.assemblyai.com/faqs/supported-file-formats) * [Transcribing dual channel/stereo recordings](https://docs.assemblyai.com/guides/transcribing-dual-channel-stereo-recordings) * [Getting speaker labels (speaker diarization)](https://docs.assemblyai.com/guides/getting-speaker-labels-speaker-diarization) diff --git a/content/posts/201009-accurate-twilio-voice-call-recording-transcriptions-assemblyai.markdown b/content/posts/201009-accurate-twilio-voice-call-recording-transcriptions-assemblyai.markdown index e178de998..a041498f9 100644 --- a/content/posts/201009-accurate-twilio-voice-call-recording-transcriptions-assemblyai.markdown +++ b/content/posts/201009-accurate-twilio-voice-call-recording-transcriptions-assemblyai.markdown @@ -14,7 +14,7 @@ is commonly used to initiate and receive phone calls, but the transcription accuracy for [recordings](https://www.twilio.com/docs/voice/api/recording) often leaves a lot to be desired. In this tutorial, we'll see how to connect an outbound phone call powered by the Twilio Voice API with -[AssemblyAI's deep learning transcription API](https://docs.assemblyai.com/guides/transcribing-an-audio-file-recording) +[AssemblyAI's deep learning transcription API](https://docs.assemblyai.com/overview/getting-started) to get significantly more accurate speech-to-text output. @@ -546,7 +546,7 @@ that needs high quality text-to-speech transcription. If the results aren't quite good enough for you, check out this tutorial on [boosting accuracy for keywords or phrases](https://docs.assemblyai.com/guides/boosting-accuracy-for-keywords-or-phrases) as well as -[selecting a model that better matches your data](https://docs.assemblyai.com/guides/transcribing-with-a-different-acoustic-or-custom-language-model). +[better matching your data with topic detection](https://docs.assemblyai.com/guides/iab-categorization). ## What now? diff --git a/content/posts/210105-django-accurate-twilio-voice-transcriptions.markdown b/content/posts/210105-django-accurate-twilio-voice-transcriptions.markdown index 94df3bed8..c4b7d0018 100644 --- a/content/posts/210105-django-accurate-twilio-voice-transcriptions.markdown +++ b/content/posts/210105-django-accurate-twilio-voice-transcriptions.markdown @@ -3,7 +3,7 @@ slug: django-accurate-twilio-voice-transcriptions meta: Use Python, Django and AssemblyAI's transcription API to improve recording accuracy for Twilio Programmable Voice phone calls. category: post date: 2021-01-05 -modified: 2021-01-05 +modified: 2021-09-13 newsletter: False headerimage: /img/headers/django-assemblyai.jpg headeralt: Logos for the implementations used in this blog post. Copyright their respective owners. @@ -692,9 +692,7 @@ The second part is just the plain text output from the transcription. You can take this now take this base code and add it to any application that needs high quality text-to-speech transcription. If the results -aren't quite good enough for you, check out this tutorial on -[selecting a model that better matches your data](https://docs.assemblyai.com/guides/transcribing-with-a-different-acoustic-or-custom-language-model) -as well as +aren't quite good enough for you yet, check out this tutorial on [boosting accuracy for keywords or phrases](https://docs.assemblyai.com/guides/boosting-accuracy-for-keywords-or-phrases). diff --git a/content/posts/210823-performance-monitoring-aws-lambda-sentry.markdown b/content/posts/210823-performance-monitoring-aws-lambda-sentry.markdown new file mode 100644 index 000000000..9ede782ef --- /dev/null +++ b/content/posts/210823-performance-monitoring-aws-lambda-sentry.markdown @@ -0,0 +1,261 @@ +title: Application Performance Monitoring AWS Lambda Functions with Sentry +slug: application-performance-monitoring-aws-lambda-functions-sentry +meta: Learn how to use Sentry Application Performance Monitoring on AWS Lambda. +category: post +date: 2021-08-23 +modified: 2021-08-26 +newsletter: False +headerimage: /img/headers/python-lambda-sentry.jpg +headeralt: The Python, AWS Lambda and Sentry logos are copyright their respective owners. + + +[Amazon Web Services (AWS) Lambda](/aws-lambda.html) is a usage-based +computing infrastructure service that can execute +[Python 3](/why-use-python.html) code. One of the challenges of this +environment is ensuring efficient performance of your Lambda Functions. +Application performance monitoring (APM) is particularly useful in these +situations because you are billed based on how long you use the +resources. + +In this post we will install and configure +[Sentry's APM](https://sentry.io/for/performance/) that works via a +[Lambda layer](https://docs.aws.amazon.com/lambda/latest/dg/invocation-layers.html). +Note that if you are looking for error monitoring rather than performance +monitoring, take a look at +[How to Monitor Python Functions on AWS Lambda with Sentry](/blog/monitor-python-functions-aws-lambda-sentry.html) +rather than following this post. + + +## First steps with AWS Lambda +A local [development environment](/development-environments.html) is not +required to follow this tutorial because all of the coding and configuration +can happen in a web browser through the +[AWS Console](https://console.aws.amazon.com/console/). + +[Sign into your existing AWS account](https://aws.amazon.com/console) +or sign up for a [new account](https://aws.amazon.com/). Lambda +gives you the first 1 million requests for free so that you can execute +basic applications without no or low cost. + +The AWS Lambda landing page. + +When you log into your account, use the search box to enter +"lambda" and select "Lambda" when it appears to get to the right +page. + +Use the search bar to find AWS Lambda. + +If you have already used Lambda before, you will see your existing Lambda +functions in a searchable table. We're going to create a new function so +click the "Create function" button. + +Click the create function button. + +The create function page will give you several options for building a +Lambda function. + +The create function details page. + +Click the "Browse Serverless App Repository" selection box, then choose +the "hello-world-python3" starter app from within the +"Public applications" section. + +The create function details page. + +The hello-world-python3 starter app details page should look something +like the following screen: + +Hello world Python3 example app and Lambda function. + +Fill in some example text such as "test" under `IdentityNameParameter` +and click the "Deploy" button: + +Click the deploy button to use the starter app. + +The function will now be deployed. As soon as it is ready we can +customize it and test it out before adding Sentry to capture any errors +that occur during execution. + +Go back to the Lambda functions main page and select your new deployed +starter app from the list. + +List of AWS Lambda functions you have created. + +Find the orange "Test" button with a down arrow next to it like you +see in the image below, and then click the down arrow. Select +"Configure Test Event". + +Configure the test event. + +Fill in the Event name as "FirstTest" or something similar, then +press the "Create" button at the bottom of the modal window. + +Click the "Test" button and it will run the Lambda function with +the parameters from that new test event. You should see something +like the following output: + +```python +Response +"value1" + +Function Logs +START RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914 Version: $LATEST +value1 = value1 +value2 = value2 +value3 = value3 +END RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914 +REPORT RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914 Duration: 0.30 ms Billed Duration: 1 ms Memory Size: 128 MB Max Memory Used: 43 MB Init Duration: 1.34 ms + +Request ID +62fa2f25-669c-47b7-b4e7-47353b0bd914 +``` + +The code was successfully executed, so let's add Sentry's performance +monitoring and test some code that uses it. + + +## Performance monitoring with Sentry +Go to [Sentry.io's homepage](https://sentry.io). + +Sentry.io homepage where you can sign up for a free account. + +Sign into your account or sign up for a new free account. You will be at +the main account dashboard after logging in or completing the Sentry sign +up process. + +Select "Performance" on the left navigation bar, it will take you to the +performance monitoring page. + +Click the 'performance' button on the left side nav. + +Click "Start Setup" then go back over to AWS Lambda to complete the +steps for adding Sentry's Python layer to your Lambda function. + +The easiest way to add Sentry to Lambda for this application +is to configure an +[AWS Lambda Layer](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html) +with the necessary dependency for Sentry. Sentry has concise +[documentation on adding via Lambda Layers](https://docs.sentry.io/platforms/python/guides/aws-lambda/layer/) +so we will walk through that way to configure it and test it +out. + +Scroll down to the "Layers" section while in your Lambda +function configuration. Click the "Add a layer" button": + +Add Lambda layer. + +In the "Add layer" screen, select the "Specify an ARN" option. + +Select Specify ARN in the Add Layer screen. + +Now to specify the Amazon Resource Name (ARN), we need to use +the Sentry documentation to get the right configuration string. + +US-East-1 is the oldest and most commonly-used region so I'll +use that here in this tutorial but you should check which one +you are in if you are not certain. + +Select the AWS for the ARN string. + +Copy that value into the Lambda Layer configuration, like this: + +Select the AWS for the ARN string. + +Then press the "Add" button. You now have the Sentry dependency +in your environment so code that relies upon that library can be +used in the Lambda function. + + +## Testing performance monitoring +Let's change our Python code in the Lambda function and test out +the APM agent. + +Make sure you are signed into your Sentry account and go to +[this specific AWS Lambda set up guide](https://docs.sentry.io/platforms/python/guides/aws-lambda/). + +You will see a "DSN string" that we need to set as an environment +variable on AWS Lambda to finish our setup. Copy the string that +matches your project as shown on that page in the highlighted green +section: + +Copy the Sentry DSN string so we can export it as an environment variable. + +We will +[use environment variables on AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html) +to store and access values like this Sentry DSN key. + +Go into the Lambda console to create a new environment variable. To do +that, click the "Configuration" tab within Lambda like you see here: + +Click the Lambda Configuration tab. + +Then click "Edit" and add a new environment variable with the key of `SENTRY_DSN` +and the value of the DSN string that you copied from the Sentry screen. + +Add the environment variable in AWS Lambda. + +Click the "Save" button and go back to your Lambda function's code editor. + +Replace the code in your Lambda function with the following code: + +```python +import json +import os +import sentry_sdk +import time +from sentry_sdk.integrations.aws_lambda import AwsLambdaIntegration +from sentry_sdk import start_transaction + +SENTRY_DSN = os.environ.get('SENTRY_DSN') +sentry_sdk.init( + dsn=SENTRY_DSN, + traces_sample_rate=1.0, + integrations=[AwsLambdaIntegration()] +) + +print('Loading function') + + +def lambda_handler(event, context): + calc = 1000 + + # this is custom instrumentation, see docs: https://bit.ly/2WjT3AY + with start_transaction(op="task", name="big calculation"): + for i in range(1, 1000): + calc = calc * i + + print(calc) + return event['key1'] # Echo back the first key value +``` + +The above code imports the Sentry dependencies, and then runs both +[automatic instrumentation](https://docs.sentry.io/platforms/python/guides/aws-lambda/performance/instrumentation/automatic-instrumentation/) +and [custom instrumentation](https://bit.ly/2WjT3AY) on the +code. Click the "Deploy" button and then "Test". The code will +successfully execute and when we go back to our Sentry performance +monitoring dashboard we will see some initial results, like this +following screenshot. + +APM results shown in the Sentry dashboard. + +Looks good, you have both the default and the specified transaction +performance recordings in the dashboard, and you can toggle between +them (or other transactions you record) through the user interface. + + +## What's Next? +We just wrote and executed a Python 3 function on AWS Lambda that +used the basics of Sentry APM to get some initial performance +monitoring data. + +Check out the [AWS Lambda section](/aws-lambda.html) for +more tutorials by other developers. + +Further questions? Contact me on Twitter +[@fullstackpython](https://twitter.com/fullstackpython) +or [@mattmakai](https://twitter.com/mattmakai). I am also on GitHub with +the username [mattmakai](https://github.com/mattmakai). + +Something wrong with this post? Fork +[this page's source on GitHub](https://github.com/mattmakai/fullstackpython.com/blob/master/content/posts/210823-performance-monitoring-aws-lambda-sentry.markdown) +and submit a pull request. diff --git a/static/img/210823-sentry-apm-lambda/arn-region.png b/static/img/210823-sentry-apm-lambda/arn-region.png new file mode 100644 index 000000000..4bca3e021 Binary files /dev/null and b/static/img/210823-sentry-apm-lambda/arn-region.png differ diff --git a/static/img/210823-sentry-apm-lambda/layer-with-arn.png b/static/img/210823-sentry-apm-lambda/layer-with-arn.png new file mode 100644 index 000000000..577cbba74 Binary files /dev/null and b/static/img/210823-sentry-apm-lambda/layer-with-arn.png differ diff --git a/static/img/210823-sentry-apm-lambda/performance-results.jpg b/static/img/210823-sentry-apm-lambda/performance-results.jpg new file mode 100644 index 000000000..0e9d3f96d Binary files /dev/null and b/static/img/210823-sentry-apm-lambda/performance-results.jpg differ diff --git a/static/img/210823-sentry-apm-lambda/performance.jpg b/static/img/210823-sentry-apm-lambda/performance.jpg new file mode 100644 index 000000000..7871121d7 Binary files /dev/null and b/static/img/210823-sentry-apm-lambda/performance.jpg differ diff --git a/static/img/210823-sentry-apm-lambda/sentry-dsn-string.png b/static/img/210823-sentry-apm-lambda/sentry-dsn-string.png new file mode 100644 index 000000000..873e68480 Binary files /dev/null and b/static/img/210823-sentry-apm-lambda/sentry-dsn-string.png differ diff --git a/static/img/fsp-fav.png b/static/img/fsp-fav.png index 28a9f6cdb..3a2f75910 100644 Binary files a/static/img/fsp-fav.png and b/static/img/fsp-fav.png differ diff --git a/static/img/logos/cloudflare.png b/static/img/logos/cloudflare.png new file mode 100644 index 000000000..b6a18d47e Binary files /dev/null and b/static/img/logos/cloudflare.png differ diff --git a/static/img/logos/oracle.jpg b/static/img/logos/oracle.jpg new file mode 100644 index 000000000..6e8fec361 Binary files /dev/null and b/static/img/logos/oracle.jpg differ diff --git a/static/img/visuals/cx-oracle.jpg b/static/img/visuals/cx-oracle.jpg new file mode 100644 index 000000000..878e7d285 Binary files /dev/null and b/static/img/visuals/cx-oracle.jpg differ diff --git a/static/img/visuals/oracle-orm-examples.jpg b/static/img/visuals/oracle-orm-examples.jpg new file mode 100644 index 000000000..adae67764 Binary files /dev/null and b/static/img/visuals/oracle-orm-examples.jpg differ diff --git a/theme/templates/base.html b/theme/templates/base.html index 6c14fbb83..93e3ad132 100644 --- a/theme/templates/base.html +++ b/theme/templates/base.html @@ -17,9 +17,8 @@ {% block content %}{% endblock %}

{% block lower_banner %}{% endblock %} -
Matt Makai 2012-2021
+
Matt Makai 2012-2022
{% block bottom_banner %}{% endblock %} - {% block js %}{% endblock %} diff --git a/theme/templates/blog/application-performance-monitoring-aws-lambda-functions-sentry.html b/theme/templates/blog/application-performance-monitoring-aws-lambda-functions-sentry.html new file mode 100644 index 000000000..e38e97585 --- /dev/null +++ b/theme/templates/blog/application-performance-monitoring-aws-lambda-functions-sentry.html @@ -0,0 +1,7 @@ + +Learning Programming +Web development +AWS Lambda +Sentry +Sentry homepage {% include "blog/external-link.html" %} +Sentry Python Quickstart docs {% include "blog/external-link.html" %} diff --git a/theme/templates/blog/sentry-application-performance-monitor-django.html b/theme/templates/blog/sentry-application-performance-monitor-django.html new file mode 100644 index 000000000..6d581b5ea --- /dev/null +++ b/theme/templates/blog/sentry-application-performance-monitor-django.html @@ -0,0 +1,6 @@ +Learning Programming +Web development +Django +Sentry +Sentry homepage {% include "blog/external-link.html" %} +Sentry Performance Monitoring docs {% include "blog/external-link.html" %} diff --git a/theme/templates/choices/demo-software-developers.html b/theme/templates/choices/demo-software-developers.html new file mode 100644 index 000000000..e69de29bb diff --git a/theme/templates/choices/oracle.html b/theme/templates/choices/oracle.html new file mode 100644 index 000000000..2ff0b969a --- /dev/null +++ b/theme/templates/choices/oracle.html @@ -0,0 +1,18 @@ +

What's next to get your app running?

+
+
+
+ {% include "choices/buttons/no-sql-datastore.html" %} +
+
+
+
+ {% include "choices/buttons/cascading-style-sheets.html" %} +
+
+
+
+ {% include "choices/buttons/javascript.html" %} +
+
+
diff --git a/theme/templates/css/base.css b/theme/templates/css/base.css index 06a5ef2ed..5a7ff8055 100644 --- a/theme/templates/css/base.css +++ b/theme/templates/css/base.css @@ -1 +1 @@ -{% raw %}hr,img{border:0}*,:after,:before{box-sizing:border-box}html{-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%;font-size:18px;background:#fefefe}body{margin:0;font:18px Georgia,serif;line-height:1.4;color:#222;padding:0}img{vertical-align:middle}hr{height:0;box-sizing:content-box;margin:21px 0;border-top:1px solid #eee}h1,h2,h3,h4,h5,h6{font-family:"Helvetica Neue",sans-serif;font-weight:500;line-height:1.1;color:#000}h1,h2,h3{margin:30px 0 6px}h1{font-size:40px}h2{font-size:28px}h3{font-size:22px}h4,h5,h6{margin:11px 0;font-size:18px}p{margin:0 0 12px}ol,ul{margin:0 0 10px}code,pre{font:"Courier New",monospace;border-radius:4px;background-color:#f4f9ff;font-size:12px}code{padding:2px 4px;white-space:nowrap}pre{overflow:scroll;white-space:pre;display:block;padding:10px;margin:0 0 11px;line-height:1.4;word-break:break-all;word-wrap:break-word;border:1px solid #ccc}.cn{padding:0 15px 0 15px;margin-right:auto;margin-left:auto}.cn:before,.cn:after{display:table;content:" "}.cn:after{clear:both}.row:before,.row:after{display:table;content:" "}.row:after{clear:both}.c2,.c3,.c4,.c5,.c6,.c7,.c8,.c9,.c10,.c11,.c12{position:relative;min-height:1px;padding:0 15px 0 15px}a{background:transparent;text-decoration:none;border-bottom:1px dotted;color:#444}a:hover{text-decoration:none;color:#000}.ft{padding:0 0 24px;float:right}.sns{font-family:"Helvetica Neue",sans-serif}.sps{font-size:14px}.hd{margin:20px 0 15px 0}.hd>a{border-bottom:none}img.hdr{vertical-align:middle;border:none;height:52px;width:52px;padding:1px}.hdt a,.hdt a:hover{font:72px "Helvetica Neue",sans-serif;font-weight:normal;letter-spacing:.03em;vertical-align:middle;margin-left:5px;color:#000;text-decoration:none;border-bottom:none;line-height:.9em}.bk{margin:0 5px 0 5px}img.nob{border:none}p.banner{font-weight:500;line-height:1.1;color:#fff;font-size:22px;margin:14px 0 18px 0}.bp{line-height:1.3em}{% endraw %} +{% raw %}hr,img{border:0}*,:after,:before{box-sizing:border-box}html{-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%;font-size:18px;background:#fefefe}body{margin:0;font:18px Georgia,serif;line-height:1.4;color:#222;padding:0}img{vertical-align:middle}hr{height:0;box-sizing:content-box;margin:21px 0;border-top:1px solid #eee}h1,h2,h3,h4,h5,h6{font-family:"Helvetica Neue",sans-serif;font-weight:500;line-height:1.1;color:#000}h1,h2,h3{margin:30px 0 6px}h1{font-size:40px}h2{font-size:28px}h3{font-size:22px}h4,h5,h6{margin:11px 0;font-size:18px}p{margin:0 0 12px}ol,ul{margin:0 0 10px}code,pre{font:"Courier New",monospace;border-radius:4px;background-color:#f4f9ff;font-size:12px}code{padding:2px 4px;white-space:nowrap}pre{overflow:scroll;white-space:pre;display:block;padding:10px;margin:0 0 11px;line-height:1.4;word-break:break-all;word-wrap:break-word;border:1px solid #ccc}.cn{padding:0 15px 0 15px;margin-right:auto;margin-left:auto}.cn:before,.cn:after{display:table;content:" "}.cn:after{clear:both}.row:before,.row:after{display:table;content:" "}.row:after{clear:both}.c2,.c3,.c4,.c5,.c6,.c7,.c8,.c9,.c10,.c11,.c12{position:relative;min-height:1px;padding:0 15px 0 15px}a{background:transparent;text-decoration:none;border-bottom:1px dotted;color:#444}a:hover{text-decoration:none;color:#000}.ft{padding:0 0 24px;float:right}.sns{font-family:"Helvetica Neue",sans-serif}.sps{font-size:14px}.hd{margin:20px 0 15px 0}.hd>a{border-bottom:none}img.hdr{vertical-align:middle;border:none;height:52px;width:52px;padding:1px}.hdt a,.hdt a:hover{font:72px "Helvetica Neue",sans-serif;font-weight:normal;letter-spacing:.03em;vertical-align:middle;margin-left:-5px;color:#000;text-decoration:none;border-bottom:none;line-height:.9em}.bk{margin:0 5px 0 5px}img.nob{border:none}p.banner{font-weight:500;line-height:1.1;color:#fff;font-size:22px;margin:14px 0 18px 0}.bp{line-height:1.3em}{% endraw %} diff --git a/theme/templates/index-sidebar.html b/theme/templates/index-sidebar.html index 5fcd2647d..ccfb5ad9e 100644 --- a/theme/templates/index-sidebar.html +++ b/theme/templates/index-sidebar.html @@ -1,3 +1,2 @@
- {% include "sponsor/sentry-assemblyai.html" %}
diff --git a/theme/templates/index.html b/theme/templates/index.html index e021e8703..3c62f365b 100644 --- a/theme/templates/index.html +++ b/theme/templates/index.html @@ -5,15 +5,15 @@ {% endblock %} -{% block css %}{% endblock %} +{% block css %}{% endblock %} {% block banner %} {% endblock %} {% block content %}
-
-

Build, Deploy and Operate Python Applications

+
+

Learn to Build, Deploy and Operate Python Applications

You're knee deep in learning Python programming. The syntax is starting to make sense. The first few ahh-ha! moments hit you as you learn to use @@ -44,7 +44,6 @@

Build, Deploy and Operate Python Applications

What do you need to learn first?

- {% include "index-sidebar.html" %}
diff --git a/theme/templates/nav.html b/theme/templates/nav.html index 529a36676..5b91021c4 100644 --- a/theme/templates/nav.html +++ b/theme/templates/nav.html @@ -1 +1 @@ -
Full Stack Python logo Full Stack Python
{% include "subnav.html" %}
+
Full Stack Python
{% include "subnav.html" %}
diff --git a/theme/templates/sponsor.html b/theme/templates/sponsor.html index 6485b9e0a..bc378aee8 100644 --- a/theme/templates/sponsor.html +++ b/theme/templates/sponsor.html @@ -1,5 +1,4 @@ {% if page.sortorder[0:2] == "01" or page.sortorder[0:2] == "02" or page.sortorder[0:2] == "03" or page.sortorder[0:2] == "04" or page.sortorder[0:2] == "05" or page.sortorder[0:2] == "06" or page.sortorder[0:2] == "50" %} -{% include "sponsor/sentry-assemblyai.html" %} {% include "sponsor/carbon.html" %} {% endif %} {% if false %} diff --git a/theme/templates/sponsor/assemblyai.html b/theme/templates/sponsor/assemblyai.html new file mode 100644 index 000000000..431a8c567 --- /dev/null +++ b/theme/templates/sponsor/assemblyai.html @@ -0,0 +1,7 @@ +
+

Sponsored By

+
+ AssemblyAI logo +

The automatic transcription API loved by Python developers.

+
+
diff --git a/theme/templates/subnav.html b/theme/templates/subnav.html index cbf4e08e9..f57d81fd8 100644 --- a/theme/templates/subnav.html +++ b/theme/templates/subnav.html @@ -1,10 +1,8 @@
All topics | Blog | - Supporter's Edition | @fullstackpython | Facebook | - What's new?