diff --git a/content/pages/01-introduction/01-learning-programming.markdown b/content/pages/01-introduction/01-learning-programming.markdown index 217546782..08f676d8e 100644 --- a/content/pages/01-introduction/01-learning-programming.markdown +++ b/content/pages/01-introduction/01-learning-programming.markdown @@ -125,11 +125,6 @@ repositories and sites with practice problems and solutions: * [TeachCraft](https://teachcraft.net/) combines Minecraft with Python to learn coding. -* [500 Data Structures and Algorithms practice problems and their solutions](https://techiedelight.quora.com/500-Data-Structures-and-Algorithms-practice-problems-and-their-solutions) - covers a large swath of the computer science space. It is not important - to know all of these algorithms and data structures but experience with - many of them will be greatly beneficial in becoming a better developer. - ### First-hand advice These articles are written by programmers who explain how they learned to @@ -145,11 +140,6 @@ give example paths you can think about taking as a beginner: including persistence, respecting others and considering ideas that are outside your comfort zone. -* [Mastering programming](https://www.facebook.com/notes/kent-beck/mastering-programming/1184427814923414) - by [Kent Beck](https://en.wikipedia.org/wiki/Kent_Beck) contains - patterns and observations for how experienced programmers he has worked - with in the past became great software developers. - * [This Picture Will Change the Way You Learn to Code](https://dev.to/nextdotxyz/this-picture-will-change-the-way-you-learn-tocode-4kmh) covers a well done graphics of many up-to-date concepts and tools that developers use. The post reminds you that you will not and should not learn @@ -170,10 +160,6 @@ your teaching experience: is an awesome resource that explains how you can use simple but fun drawings to teach otherwise difficult technical concepts to students. -* [Teaching programming to working professionals](http://pgbovine.net/PG-Podcast-21-Trey-Hunner.htm) - is a video podcast with [Trey Hunter](https://twitter.com/treyhunner) - about his experience teaching Python to experienced professionals. - * [Teach Yourself Computer Science](https://teachyourselfcs.com/) is intended as a self-teaching tool with many resources that are classic computer science textbooks. There are also nice explanations for why diff --git a/content/pages/01-introduction/02-python-programming-language.markdown b/content/pages/01-introduction/02-python-programming-language.markdown index 87a44028b..a3b046bb1 100644 --- a/content/pages/01-introduction/02-python-programming-language.markdown +++ b/content/pages/01-introduction/02-python-programming-language.markdown @@ -79,9 +79,6 @@ Dictionary comprehension: ### General Python language resources -* The [online Python tutor](http://www.pythontutor.com/) visually walks - through code and shows how it executes on the Python interpreter. - * [Python Module of the Week](http://pymotw.com/2/index.html) is a tour through the Python standard library. @@ -108,9 +105,6 @@ Dictionary comprehension: * Armin Roacher presented [things you didn't know about Python](https://speakerdeck.com/mitsuhiko/didntknow) at PyCon South Africa in 2012. -* [Writing idiomatic Python](http://www.jeffknupp.com/blog/2012/10/04/writing-idiomatic-python/) - is a guide for writing Pythonic code. - ### Python ecosystem resources There's an entire page on [best Python resources](/best-python-resources.html) @@ -167,10 +161,6 @@ the very beginner topics. covers what the code for list comprehensions looks like and gives some example code to show how they work. -* [An Introduction to Python Lists](http://effbot.org/zone/python-list.htm) - is a solid overview of Python lists in general and tangentially covers - list comprehensions. - ### Python generator resources * This blog post entitled diff --git a/content/pages/01-introduction/03-why-use-python.markdown b/content/pages/01-introduction/03-why-use-python.markdown index 505585cf3..86a0e3ecd 100644 --- a/content/pages/01-introduction/03-why-use-python.markdown +++ b/content/pages/01-introduction/03-why-use-python.markdown @@ -33,10 +33,7 @@ The IEEE ranked Python as the [#1 programming language in 2019](https://spectrum.ieee.org/computing/software/the-top-programming-languages-2019), which continued its hot streak after ranking it -[#1 in 2018](https://spectrum.ieee.org/at-work/innovation/the-2018-top-programming-languages), -[#1 in 2017](https://spectrum.ieee.org/at-work/innovation/the-2017-top-programming-languages) -and -[#3 top programming language in 2016](http://spectrum.ieee.org/computing/software/the-2016-top-programming-languages). +[#1 in 2018](https://spectrum.ieee.org/at-work/innovation/the-2018-top-programming-languages) and 2017. [RedMonk's June 2019 ranking](https://redmonk.com/sogrady/2019/07/18/language-rankings-6-19/) had Python at #3, which held consistent from previous years' rankings in [2018](https://redmonk.com/sogrady/2018/08/10/language-rankings-6-18/) @@ -133,13 +130,6 @@ language. system versus statically typed languages, be sure to [read this thorough explanation of the topic](http://blogs.perl.org/users/ovid/2010/08/what-to-know-before-debating-type-systems.html). -* [Why I swapped C#.NET for Python as my default language and platform (and won’t be going back)](https://medium.com/@anthonypjshaw/why-i-swapped-c-net-for-python-as-my-default-language-and-platform-and-wont-be-going-back-e0063a25e491) - provides a viewpoint from someone who is not a professional developer but - uses coding to hack out some projects. He found Microsoft's .NET ecosystem - lacking when it came to satisfying his needs and Python filled the gap for - him with its wide array of open source code libraries, package management - and ability to work well on platforms other than Windows. - * [Python, Machine Learning, and Language Wars](http://sebastianraschka.com/blog/2015/why-python.html) compares Python with R, MATLAB and Julia for data science work. While Python is great for [deployment automation](/deployment.html) and diff --git a/content/pages/01-introduction/04-python-2-or-3.markdown b/content/pages/01-introduction/04-python-2-or-3.markdown index eaf166982..a362e7b49 100644 --- a/content/pages/01-introduction/04-python-2-or-3.markdown +++ b/content/pages/01-introduction/04-python-2-or-3.markdown @@ -78,7 +78,7 @@ gone through the process and have advice for making it less painful. implementations. There is also a [quick reference for writting code with Python 2 and 3 compatibility](https://wiki.python.org/moin/PortingToPy3k/BilingualQuickRef). -* [Upgrading to Python 3 with Zero Downtime](https://tech.yplanapp.com/2016/08/24/upgrading-to-python-3-with-zero-downtime/) +* [Upgrading to Python 3 with Zero Downtime](https://adamj.eu/tech/2016/08/24/upgrading-yplan-to-python-3-with-zero-downtime/) supplies advice on transitioning a large existing Python 2 web application to Python 3. Their process involved upgrading dependencies, testing and deploying the new version before going back to clean up unnecessary code diff --git a/content/pages/01-introduction/05-enterprise-python.markdown b/content/pages/01-introduction/05-enterprise-python.markdown index d00e8fbe0..c50052847 100644 --- a/content/pages/01-introduction/05-enterprise-python.markdown +++ b/content/pages/01-introduction/05-enterprise-python.markdown @@ -39,12 +39,14 @@ frameworks when otherwise they should not make technical design decisions. ## Why are there misconceptions about Python in enterprise environments? Traditionally large organizations building enterprise software have used -statically typed languages such as C++, .NET and Java. Throughout the 1980s -and 1990s large companies such as Microsoft, Sun Microsystems and Oracle -marketed these languages as "enterprise grade". The inherent snub to other -languages was that they were not appropriate for CIOs' difficult technical -environments. Languages other than Java, C++ and .NET were seen as risky and -therefore not worthy of investment. +statically typed languages and platforms such as C++, C# and Java. +Throughout the 1990s and early 2000s, large companies such as +Microsoft, Sun Microsystems and Oracle marketed these languages as +"enterprise grade". The inherent message about other programming +ecosystem was that they were not appropriate for CIOs' difficult +technical environments. Languages other than Java, C++ and C# (along +with its broader .NET platform) were seen as risky and therefore not +worthy of investment. In addition, "scripting languages" such as Python, Perl and Ruby were not yet robust enough in the 1990s because their core standard libraries were @@ -68,7 +70,7 @@ best maintained and fully featured pieces of code for any language. Meanwhile, some of the traditional enterprise software development languages such as Java have languished due to underinvestment by their major corporate -backers. When [Oracle purchased Sun Microsystems in 2009](http://www.oracle.com/us/corporate/press/018363) +backers. When Oracle purchased Sun Microsystems in 2009, there was a long lag time before Java was enhanced with new language features in Java 7. Oracle also [bundles unwanted adware with the Java installation](http://www.engadget.com/2015/03/06/java-adware-mac/), diff --git a/content/pages/01-introduction/06-community.markdown b/content/pages/01-introduction/06-community.markdown index 1baec5f4f..a5f463319 100644 --- a/content/pages/01-introduction/06-community.markdown +++ b/content/pages/01-introduction/06-community.markdown @@ -77,7 +77,7 @@ resources provide perspective on offline events like and [PyCon US 2016](http://www.dreisbach.us/blog/pycon-2016/). There are many other retrospectives for other - [community-led conferences such as EuroPython](http://www.artima.com/weblogs/viewpost.jsp?thread=261930). + [community-led conferences such as EuroPython](https://www.artima.com/weblogs/viewpost.jsp?thread=261930). These summaries can be a great way to get a slice of the experience before purchasing a ticket and booking a trip. @@ -112,16 +112,11 @@ the community. provides a starter page with links to community-run newsletters, resources and conferences. -* There are many large active online communities on - [Reddit](https://www.reddit.com/r/python) and - [IRC channels](https://freenode.net/) such as #python, #python-dev - and #distutils. - * The Python community has a concept known as "Benevolent Dictator For Life" that may appear odd to newcomers. Essentially, Guido Van Rossum created the language and still has the ability to decide community arguments one way or the other. This post on the - [origin of BDFL](http://www.artima.com/weblogs/viewpost.jsp?thread=235725) + [origin of BDFL](https://www.artima.com/weblogs/viewpost.jsp?thread=235725) has more context about Guido's role. * [Python Community and Python at Dropbox](https://talkpython.fm/episodes/show/30/python-community-and-python-at-dropbox) diff --git a/content/pages/01-introduction/07-companies.markdown b/content/pages/01-introduction/07-companies.markdown index 7eb7a78f2..c4e02cfc8 100644 --- a/content/pages/01-introduction/07-companies.markdown +++ b/content/pages/01-introduction/07-companies.markdown @@ -37,7 +37,7 @@ below). at job descriptions on sites like [Glassdoor with "Python Goldman Sachs" keywords](https://www.glassdoor.com/Jobs/Goldman-Sachs-python-Jobs-EI_IE2800.0,13_KO14,20.htm) and - [Indeed for JP Morgan Chase](https://www.indeed.com/salaries/Python-Developer-Salaries-at-JPMorgan-Chase). + [Indeed for JP Morgan Chase](https://www.indeed.com/cmp/JPMorgan-Chase/salaries/Python-Developer). Salaries and responsibilities vary widely based on the role and whether Python is used for data analysis, [web application development](/web-development.html) or DevOps. diff --git a/content/pages/01-introduction/08-best-python-resources.markdown b/content/pages/01-introduction/08-best-python-resources.markdown index 81113487a..6677ba3d2 100644 --- a/content/pages/01-introduction/08-best-python-resources.markdown +++ b/content/pages/01-introduction/08-best-python-resources.markdown @@ -47,13 +47,6 @@ should skip down to the next section for "experienced developers". and problems rather than jumping into a specific language that's recommended to you by a friend. -* [A Python Crash Course](https://www.grahamwheeler.com/posts/python-crash-course.html) - gives an awesome overview of the history of Python, what drives the - programming community and dives into example code. You will likely need - to read this in combination with other resources to really let the syntax - sink in, but it's a great article to read several times over as you - continue to learn. - * The [Python projects tag](https://www.twilio.com/blog/tag/python) on the Twilio blog is constantly updated with fun tutorials you can build to learn Python, such as the @@ -61,9 +54,6 @@ should skip down to the next section for "experienced developers". [Choose Your Own Adventures Presentations using Flask and WebSockets](https://www.twilio.com/blog/2014/11/choose-your-own-adventure-presentations-with-reveal-js-python-and-websockets.html) and [Martianify Photos with OpenCV](https://www.twilio.com/blog/2015/11/getting-started-with-opencv-and-python-featuring-the-martian-2.html). -* [A Byte of Python](http://www.swaroopch.com/notes/python/) is a beginner's - tutorial for the Python language. - * Google put together a great compilation of materials and subjects you should read and learn from if you want to be a [professional programmer](https://www.google.com/about/careers/students/guide-to-technical-development.html). diff --git a/content/pages/01-introduction/09-best-python-videos.markdown b/content/pages/01-introduction/09-best-python-videos.markdown index b265bc34b..14be9bccd 100644 --- a/content/pages/01-introduction/09-best-python-videos.markdown +++ b/content/pages/01-introduction/09-best-python-videos.markdown @@ -163,6 +163,7 @@ like [PyCon US](https://us.pycon.org/) and Python language. * PyCon US videos from + [2020](https://www.youtube.com/playlist?list=PL2Uw4_HvXqvbpFquYIE57BEAqkQWk-iFg), [2019](https://www.youtube.com/channel/UCxs2IIVXaEHHA4BtTiWZ2mQ/videos), [2018](https://www.youtube.com/channel/UCsX05-2sVSH7Nx3zuk3NYuQ/videos), [2017](https://www.youtube.com/channel/UCrJhliKNQ8g0qoE_zvL8eVg/videos), @@ -172,6 +173,7 @@ like [PyCon US](https://us.pycon.org/) and are all available online for free. * All of the talk videos are available on YouTube for + [EuroPython 2020](https://www.youtube.com/playlist?list=PL8uoeex94UhHgMD9GOCbEHWku7pEPx9fW0), [EuroPython 2019](https://www.youtube.com/playlist?list=PL8uoeex94UhHFRew8gzfFJHIpRFWyY4YW), [EuroPython 2018](https://www.youtube.com/watch?v=LoRq9yGeBWY&list=PL8uoeex94UhFrNUV2m5MigREebUms39U5), [EuroPython 2017](https://www.youtube.com/watch?v=OCHrzW-R3QI&list=PL8uoeex94UhG9QAoRICebFpeKK2M0Herh), diff --git a/content/pages/01-introduction/10-best-python-podcasts.markdown b/content/pages/01-introduction/10-best-python-podcasts.markdown index f0181a269..a2c528fee 100644 --- a/content/pages/01-introduction/10-best-python-podcasts.markdown +++ b/content/pages/01-introduction/10-best-python-podcasts.markdown @@ -42,16 +42,16 @@ listen and learn. [web application development](/web-development.html) in Python using the [Django web framework](/django.html). +* [The Real Python Podcast](https://realpython.com/podcasts/rpp/) is a weekly + podcast with interviews, coding tips, and conversation with guests from the + Python community. + * [Teaching Python](https://www.teachingpython.fm/) is a podcast by two teachers about their adventures teaching middle school computer science, problem solving, handling failure, frustration, and success with teaching Python programming. -* Professor Philip Guo has a video podcast called - [PG Podcast](https://podcasts.apple.com/us/podcast/philip-guo-podcasts-pgbovine-net/id1276072242), - which typically covers Python subjects. -* [The Real Python Podcast](https://realpython.com/podcasts/rpp/) is a weekly podcast with interviews, coding tips, and conversation with guests from the Python community. ## Favorite podcast episodes Here are a list of my favorite episodes from various Python podcasts before diff --git a/content/pages/02-development-environments/00-development-environments.markdown b/content/pages/02-development-environments/00-development-environments.markdown index cb27c7ee4..4b7dc8b94 100644 --- a/content/pages/02-development-environments/00-development-environments.markdown +++ b/content/pages/02-development-environments/00-development-environments.markdown @@ -121,12 +121,6 @@ configuration as a starting point and customize it from there. [which version of Python to use](/python-2-or-3.html) and adding [code metrics](/code-metrics.html) libraries for checking syntax. -* [Three Ways to Install Python on your Windows Computer](http://blog.yhat.com/posts/installing-python-on-windows.html) - provides multiple avenues for Windows users to get Python on their machine - before setting up the rest of their development environment. Unlike - macOS and Linux, the Windows [operating system](/operating-systems.html) - does not include Python with its default installation. - * [PyCharm: The Good Parts](http://nafiulis.me/pycharm-the-good-parts-i.html) shows you how to be more efficient and productive with that IDE if it's your choice for writing Python code. diff --git a/content/pages/02-development-environments/02-vim.markdown b/content/pages/02-development-environments/02-vim.markdown index da89bbe3d..f37c04a5d 100644 --- a/content/pages/02-development-environments/02-vim.markdown +++ b/content/pages/02-development-environments/02-vim.markdown @@ -91,6 +91,31 @@ If a Vimrc file does not already exist, just create it within the user's home directory and it will be picked up by Vim the next time you open the editor. +The following are a few resources for learning what to put in, and how to +structure a `.vimrc` file. I recommend adding configuration options one +at a time to test them individually instead of going whole hog with a Vimrc +you are unfamiliar with. + +* [A Good Vimrc](http://dougblack.io/words/a-good-vimrc.html) is a fantastic, + detailed overview and opinionated guide to configuring Vim. Highly + recommended for new and experienced Vim users. + +* [5 lines for a blank .vimrc](https://swordandsignals.com/2020/12/13/5-lines-in-vimrc.html) + shows settings for case insensitive search, highlighting as you search, + disabling swap, and a couple more "must have" enhancements to the + default configuration. + +* [Vim and Python](https://justin.abrah.ms/vim/vim_and_python.html) shows + and explains many Python-specific .vimrc options. + +* This + [repository's folder with Vimrc files](https://github.com/amix/vimrc/tree/master/vimrcs) + has example configurations that are well commented and easy to learn from. + +* For people who are having trouble getting started with Vim, check out this + blog post on the + [two simple steps that helped this author learn Vim](http://adamdelong.com/two-simple-steps-helped-me-learn-vim/). + ### Vim tutorials Vim has a reputation for a difficult learning curve, but it's much easier @@ -142,11 +167,6 @@ to get started with these tutorials. * [Vim Adventures](http://vim-adventures.com/) is a cute, fun browser-based game that helps you learn Vim commands by playing through the adventure. -* In [Vim: revisited](http://mislav.uniqpath.com/2011/12/vim-revisited/) the - author explains his on-again off-again relationship with using Vim. He then - shows how he configures and uses the editor so it sticks as his primary - code editing tool. - * [Things About Vim I Wish I Knew Earlier](https://blog.petrzemek.net/2016/04/06/things-about-vim-i-wish-i-knew-earlier/) explores the lessons one developer learned while exclusively using Vim for several years. The author includes using relative instead of absolute @@ -159,31 +179,6 @@ to get started with these tutorials. habits. -### Vimrc resources -These are a few resources for learning how to structure a `.vimrc` file. I -recommend adding configuration options one at a time to test them -individually instead of going whole hog with a Vimrc you are unfamiliar with. - -* [A Good Vimrc](http://dougblack.io/words/a-good-vimrc.html) is a fantastic, - detailed overview and opinionated guide to configuring Vim. Highly - recommended for new and experienced Vim users. - -* [Vim and Python](https://justin.abrah.ms/vim/vim_and_python.html) shows - and explains many Python-specific .vimrc options. - -* [Vim as a Python IDE](http://liuchengxu.org/posts/use-vim-as-a-python-ide/) - shows a slew of plugins and configuration options for coding with Python - in Vim. - -* This - [repository's folder with Vimrc files](https://github.com/amix/vimrc/tree/master/vimrcs) - has example configurations that are well commented and easy to learn from. - -* For people who are having trouble getting started with Vim, check out this - blog post on the - [two simple steps that helped this author learn Vim](http://adamdelong.com/two-simple-steps-helped-me-learn-vim/). - - ### Vim installation guides These installation guides will help you get Vim up and running on Mac OS X, Linux and Windows. diff --git a/content/pages/02-development-environments/04-sublime-text.markdown b/content/pages/02-development-environments/04-sublime-text.markdown index 841687737..196df3b51 100644 --- a/content/pages/02-development-environments/04-sublime-text.markdown +++ b/content/pages/02-development-environments/04-sublime-text.markdown @@ -68,10 +68,6 @@ links should get your editor customized with linters, is a spectacular tutorial that covers installing Sublime Text and configuring a multitude of helpful Python programming plugins. -* [Sublime Text 3 Heaven](https://www.kennethreitz.org/essays/sublime-text-3-heaven) - is a quick overview of the extensions, packages and bonus toys that - one developer uses for his own Sublime Text development setup. - * [Sublime Tutor](https://sublimetutor.com/) is an interactive in-editor keyboard shortcuts tutorial that plugs into Sublime so you can learn and become more productive as you use the editor. diff --git a/content/pages/02-development-environments/05-pycharm.markdown b/content/pages/02-development-environments/05-pycharm.markdown index c6810f26c..c464d6e87 100644 --- a/content/pages/02-development-environments/05-pycharm.markdown +++ b/content/pages/02-development-environments/05-pycharm.markdown @@ -29,10 +29,6 @@ Python code. is a solid discussion thread with different developers' perspectives on using PyCharm for coding their applications. -* [How to Get Started with PyCharm and Have a Productive Python IDE](https://pedrokroger.net/getting-started-pycharm-python-ide/) - covers the basics of configuring PyCharm for running code within - virtualenvs, macros, using the console and code completion. - * [Using PyCharm with Pyramid](https://docs.pylonsproject.org/projects/pyramid-cookbook/en/latest/development_tools/pycharm.html) is specific to developing and debugging [with the Pyramid web framework](/pyramid.html). diff --git a/content/pages/02-development-environments/06-jupyter-notebook.markdown b/content/pages/02-development-environments/06-jupyter-notebook.markdown index 5b2b44a06..58c9c32b6 100644 --- a/content/pages/02-development-environments/06-jupyter-notebook.markdown +++ b/content/pages/02-development-environments/06-jupyter-notebook.markdown @@ -146,6 +146,13 @@ like advanced interactive visualizations. code that is suitable for [deployment](/deployment.html) to a production environment. +* [How to use ipywidgets to make your Jupyter notebook interactive](https://www.wrighters.io/use-ipywidgets-with-jupyter-notebooks/) + is a tutorial on how to use the + [ipywidgets](https://ipywidgets.readthedocs.io/en/latest/) library + to make Jupyter Notebooks respond to users' input and go beyond + simply presenting data into having users be able to do some additional + analysis themselves. + * [28 Jupyter Notebook tips, tricks and shortcuts](https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/) explains many of the lesser-known keyboard shortcuts and mechanisms to output settings. diff --git a/content/pages/02-development-environments/08-bash-shell.markdown b/content/pages/02-development-environments/08-bash-shell.markdown index abf95629b..22c7877bc 100644 --- a/content/pages/02-development-environments/08-bash-shell.markdown +++ b/content/pages/02-development-environments/08-bash-shell.markdown @@ -20,12 +20,150 @@ during Python software development as part of a programmer's
Bash is an implementation of the shells concept. Learn more in the development environments chapter or view the table of contents for all topics.
-### Bash resources +### How do Python developers use Bash? +If you are programming in the terminal on [macOS](/macos.html) +or [Linux](/ubuntu.html), or using the +[Windows Subsystem for Linux on Windows 10](https://docs.microsoft.com/en-us/windows/wsl/install-win10), +you can easily gain access to Bash if it is not already +your default shell. + +You can show what shell you are currently using by echoing the +`SHELL` environment variable, like so: + +``` +$ echo "$SHELL" +``` + +Which will then print the shell you are currently using. For example, +on macOS I am using Bash by default so the echo command prints: + +``` +/bin/bash +``` + +How much you use Bash or any shell will likely depend on your +[development environment](/development-environments.html), especially +if you are using an editor like [Vim](/vim.html) instead of an +IDE like [PyCharm](/pycharm.html), because it is often easier to do +certain tasks in the shell. For example, most developers I know who +use PyCharm will search for some instance of source code right in +their IDE, whereas I use a combination of Vim and [tmux](/tmux.html) +so I frequently flip between panes to use commands like `grep` to +do my source code searches. + +There is no right way to perform a task like source code searching, it's +really just what works for your brain as a developer that will guide +how often you interact with the Bash shell. + + +### Getting started with Bash +Working with a shell, Bash or otherwise, is intimidating the first time +you try to get started. You are staring at the `$` prompt without a +whole lot of direction. + +When you are completely new to using Bash, it is a good idea to at least +scan, if not take some additional time for in-depth reading of the +documentation for commands that every developer uses. The following +commands are used so frequently in Bash that an experienced developer +probably does not even think about them anymore, they become just a +natural part of your workflow: + +* `echo`: [print text to the command line](https://man7.org/linux/man-pages/man1/echo.1.html) +* `ls`: [list the contents of a directory](https://man7.org/linux/man-pages/man1/ls.1.html) +* `cd`: [change the working directory](https://man7.org/linux/man-pages/man1/cd.1p.html) +* `cp`: [copy a file or directory](https://man7.org/linux/man-pages/man1/cp.1.html) +* `mv`: [move one or more files](https://man7.org/linux/man-pages/man1/mv.1.html) +* `rm`: [delete one or more files or directories](https://man7.org/linux/man-pages/man1/rm.1.html) + +If you know how to use the above commands then you will at least be able +to move around the file system, create, move and update files and know +what is on your storage device(s). + +The following commands are somewhat more advanced but also frequently +used by developers: + +* `su`: [run comamnds as different users or groups](https://man7.org/linux/man-pages/man1/su.1.html) +* `whoami`: [print which user you are currently logged in as](https://man7.org/linux/man-pages/man1/whoami.1.html) +* `grep`: [searches for patterns in files](https://man7.org/linux/man-pages/man1/grep.1.html) + +The above lists are not even close to exhaustive for what commands +you need to know when working with Bash. Read some of the following +introductory tutorials to gain a better understanding of working +with this shell: + +* [The Linux command line for beginner](https://ubuntu.com/tutorials/command-line-for-beginners) + by [Ubuntu](/ubuntu.html) will provide you with context for how to + use the command line, working with files and directories, and handling + superuser commands. + * [Bash Guide for beginners](http://www.tldp.org/LDP/Bash-Beginners-Guide/html/Bash-Beginners-Guide.html) - is an entire book for those new to working with commandlines. It covers + is an entire book for those new to working with command lines. It covers commands, paths, Bash shell scripting, variables and many other critical topics that are necessary to move from beginner to advanced Bash user. +* [101 Bash Commands and Tips for Beginners to Experts](https://dev.to/awwsmm/101-bash-commands-and-tips-for-beginners-to-experts-30je) + gives a well-done laundry list of tricks to explore. + +* [Bash Quick References](https://shellmagic.xyz/) is a cheat sheet for + common operators and signals that come up when working with scripts. + + +### Bash scripting +Bash is used not only as an interactive prompt but also for scripting, which +makes it possible to execute one or more Bash commands stored within a file. +These scripts can be short, with only a single command, or very complicated +with control-flow logic, for loops, and almost anything you want to automate +or compute because +[Bash is a Turing-complete programming language](https://www.quora.com/Is-Bash-Turing-complete). + +Complex Bash scripts sometimes get a negative reputation because they can be +difficult to read and understand if you are not the original author (or you +are reading your own script after a significant period of time has elapsed). +There are many ways to accomplish the same tasks with Bash so the files are +often confusing to read unless the author of a script included clear +documentation. This readability problem is typically less of an issue with +Python scripts because spacing is enforced and the standard library +encapsulates common tasks. + +It's a good idea to think about how you want to structure your Bash scripts +as they grow larger. The following resources provide insight into what you +should consider while coding Bash scripts. + +* This [minimal safe Bash template](https://betterdev.blog/minimal-safe-bash-script-template/) + contains an 86-line Bash script that the author claims once you + understand and use it as a base then it will make your scripts + easier to maintain over time. + +* [Creating a bash completion script](https://iridakos.com/tutorials/2018/03/01/bash-programmable-completion-tutorial.html) + is a great tutorial that walks you through a reasonably complex Bash + script for completing syntax in other Bash shell scripts. + +* [Anybody can write good bash (with a little effort)](https://blog.yossarian.net/2020/01/23/Anybody-can-write-good-bash-with-a-little-effort) + covers the basics of shell scripting and provides some recommendations + for creating more maintainable scripts such as using linters and + formatters. + +* Google's [Shell Style Guide](https://google.github.io/styleguide/shell.xml) + covers how to write consistent, maintainable shell scripts, which is + particularly important if you have ever tried to debug a hacky shell + script that was never meant to be used by anyone other than the original + author. + +* [Bash scripting quirks & safety tips](https://jvns.ca/blog/2017/03/26/bash-quirks/) + explains Bash basic programming constructs like `for` loops and variable + assignment then goes into ways to avoid weird issues in your code. + +* If all else fails when you're trying to use Bash scripts, this article + on [replacing Bash scripts with Python](https://github.com/ninjaaron/replacing-bash-scripting-with-python) + is a guide on swapping in Python for administrative scripting, including + what to do about replacing invaluable command line tools such as `awk`, + `sed` and `grep`. + + +### Additional Bash resources +The following resources cover more advanced Bash use cases and what pitfalls +to try to avoid as you work with the shell or write scripts. + * [Advancing in the Bash shell](http://samrowe.com/wordpress/advancing-in-the-bash-shell/) covers important concepts such as bang syntax, movement commands, tab completion and aliases. @@ -46,19 +184,6 @@ during Python software development as part of a programmer's `PROMPT_COMMAND`, `CDPATH` and `REPLY` which can simplify your scripts by using values that Bash already has stored for you. -* Google's [Shell Style Guide](https://google.github.io/styleguide/shell.xml) - covers how to write consistent, maintainable shell scripts, which is - particularly important if you have ever tried to debug a hacky shell - script that was never meant to be used by anyone other than the original - author. - -* [101 Bash Commands and Tips for Beginners to Experts](https://dev.to/awwsmm/101-bash-commands-and-tips-for-beginners-to-experts-30je) - is a well-done laundry list of tricks to explore. - -* [Bash scripting quirks & safety tips](https://jvns.ca/blog/2017/03/26/bash-quirks/) - explains Bash basic programming constructs like `for` loops and variable - assignment then goes into ways to avoid weird issues in your code. - * [Safe ways to do things in bash](https://github.com/anordal/shellharden/blob/master/how_to_do_things_safely_in_bash.md) shows you how to not shoot yourself in the foot by using safe coding practices with your shell scripts. @@ -76,19 +201,10 @@ during Python software development as part of a programmer's application that generates a hostable, customizable status page for your services. -* [Replacing Bash scripts with Python](https://github.com/ninjaaron/replacing-bash-scripting-with-python) - is a guide on using using Python for administrative scripting, including - what to do about replacing invaluable command line tools such as `awk`, - `sed` and `grep`. - * [Using Aliases to Speed Up Your Git Workflow](https://dev.to/robertcoopercode/using-aliases-to-speed-up-your-git-workflow-2f5a) has a bunch of shell aliases that make it easier for you to execute complicated or uncommon [Git](/git.html) commands. -* [Creating a bash completion script](https://iridakos.com/tutorials/2018/03/01/bash-programmable-completion-tutorial.html) - is a great tutorial that walks you through a reasonably complex Bash - script for completing syntax in other Bash shell scripts. - * [6 Tips Before You Write Your Next Bash Cronjob](https://yasoob.me/posts/6-tips-before-you-write-your-next-bash-cronjob/) covers starting your scripts with shebang, redirecting output, timeouts and sudo privileges. @@ -103,10 +219,15 @@ during Python software development as part of a programmer's their destructive abilities by reading through the descriptions provided by the author. -* [Bash Quick References](https://shellmagic.xyz/) is a cheat sheet for - common operators and signals that come up when working with scripts. - -* [Anybody can write good bash (with a little effort)](https://blog.yossarian.net/2020/01/23/Anybody-can-write-good-bash-with-a-little-effort) - covers the basics of shell scripting and provides some recommendations - for creating more maintainable scripts such as using linters and - formatters. +* [Faster bash startup](https://danpker.com/posts/faster-bash-startup/) + and + [Even faster bash startup](https://work.lisk.in/2020/11/20/even-faster-bash-startup.html) + are two great tutorials that will save you a bunch of time if you frequently + open new Bash shells. On many systems you can easily cut down the startup + time for the shell which can be unnecessarily sluggish. + +* [Bash HTTP monitoring dashboard](https://raymii.org/s/software/Bash_HTTP_Monitoring_Dashboard.html) + ([source code](https://github.com/RaymiiOrg/bash-http-monitoring)) + is a useful application fully written in Bash shell scripts that + monitors the health of one or more websites to make sure they are + up and running. diff --git a/content/pages/02-development-environments/12-tmux.markdown b/content/pages/02-development-environments/12-tmux.markdown index 4da1c3016..4b95fba6c 100644 --- a/content/pages/02-development-environments/12-tmux.markdown +++ b/content/pages/02-development-environments/12-tmux.markdown @@ -28,14 +28,16 @@ easier to use many shells at once and attaching to both local and remote tmux+Vim user myself, I can attest to how great these two tools complement each other. +* [Writing & Coding Workflow](http://jacobzelko.com/workflow/) shows one + developer's configuration that combines [Vim](/vim.html) and several plugins + with tmux for a productive setup. + * [Making tmux Pretty and Usable - A Guide to Customizing your tmux.conf](http://www.hamvocke.com/blog/a-guide-to-customizing-your-tmux-conf/) * [Tmux Pairing Anywhere: On Your Box](http://iamvery.com/2013/11/16/tmux-pairing-anywhere-on-your-box.html) * [Using tmux Properly](http://danielallendeutsch.com/blog/16-using-tmux-properly.html) -* [Differences between tmux vs screen](https://wtanaka.com/node/8136) - * [The Power Of tmux Hooks](https://devel.tech/tips/n/tMuXz2lj/the-power-of-tmux-hooks/) * There are a slew of "cheat sheets" for tmux out there, here are a few diff --git a/content/pages/02-development-environments/14-environment-configuration.markdown b/content/pages/02-development-environments/14-environment-configuration.markdown index 2c87e88fb..d65114eb7 100644 --- a/content/pages/02-development-environments/14-environment-configuration.markdown +++ b/content/pages/02-development-environments/14-environment-configuration.markdown @@ -34,6 +34,13 @@ Either one can be used in your applications but there are slight differences that can make one better than the other in various situations. Other useful environment variables resources: +* [How to Set Environment Variables in Linux and Mac: The Missing Manual](https://doppler.com/blog/how-to-set-environment-variables-in-linux-and-mac) + is a wonderfully detailed guide with many tips and tricks throughout + the walkthrough such as quickly setting environment variables for a + single command, passing environment variables through when using sudo + and executing a command in a "clean" environment without everything + you have already set interfering or being accessible to that script. + * [The Twelve-Factor App](https://12factor.net/) describes a method for securing environment data for your applications. The twelve factors are commonly referenced across many programming ecosystems, not just Python, diff --git a/content/pages/02-development-environments/15-application-dependencies.markdown b/content/pages/02-development-environments/15-application-dependencies.markdown index a01f2eba9..927a3720d 100644 --- a/content/pages/02-development-environments/15-application-dependencies.markdown +++ b/content/pages/02-development-environments/15-application-dependencies.markdown @@ -39,9 +39,6 @@ all yourself. A few of the best collections of Python libraries are shows the open source Python projects trending today, this week, and this month. -* This list of [20 Python libraries you can’t live without](http://freepythontips.wordpress.com/2013/07/30/20-python-libraries-you-cant-live-without/) - is a wide-ranging collection from data analysis to testing tools. - * Wikipedia actually has an extensive [page dedicated to Python libraries](http://en.wikipedia.org/wiki/List_of_Python_software) grouped by categories. @@ -161,16 +158,6 @@ so far to get up to speed on building and releasing your own packages. provides a collection of resources to understand how to package and distribute Python code libraries. -* [Alice in Python projectland](https://veekaybee.github.io/2017/09/26/python-packaging/) - is an amazing post that takes the reader from simple Python script - into a complete Python package. - -* [Perils of packaging](https://malramsay.com/post/perils_of_packaging/) - covers several edge cases that come up when trying to put together - pieces like Travis CI, PyPI and conda. The post walks through the errors - and how to get around them until they are smoothed out by updates to the - tools. - * [How to Publish Your Package on PyPI](https://blog.jetbrains.com/pycharm/2017/05/how-to-publish-your-package-on-pypi/) is for developers who have created a code library they would like to share and make installable for other developers. @@ -230,10 +217,6 @@ developers. dependency management tools including newer ones such as pipenv and Poetry. -* Occasionally arguments about using Python's dependency manager versus - one of Linux's dependency managers comes up. This provides - [one perspective on that debate](http://notes.pault.ag/debian-python/). - * [Open source trust scaling](http://lucumr.pocoo.org/2016/3/24/open-source-trust-scaling/) is a good piece for the [Python community](/python-community.html) (and other programming communities) that is based on the diff --git a/content/pages/02-development-environments/18-source-control.markdown b/content/pages/02-development-environments/18-source-control.markdown index 83913183d..c96745a51 100644 --- a/content/pages/02-development-environments/18-source-control.markdown +++ b/content/pages/02-development-environments/18-source-control.markdown @@ -194,9 +194,9 @@ implementation. free online version of the O'Reilly [Version Control with Subversion](https://www.amazon.com/dp/B002SR2QIW/) book. -* [How to use Subversion (SVN)](https://deveo.com/svn-tutorial/) lays out - the basic concepts and provides the first few steps for getting started - tracking files. +* [How to Host SVN Repositories](https://www.perforce.com/blog/vcs/how-host-subversion-svn) + lays out the basic concepts and provides the first few steps for getting + started tracking files. * [10 Most Used SVN Commands with Examples](http://www.thegeekstuff.com/2011/04/svn-command-examples/) is a good refresher list if you've used SVN in the past but it has been diff --git a/content/pages/02-development-environments/19-git.markdown b/content/pages/02-development-environments/19-git.markdown index ce70f3b16..5bffca262 100644 --- a/content/pages/02-development-environments/19-git.markdown +++ b/content/pages/02-development-environments/19-git.markdown @@ -174,10 +174,6 @@ workflow. These resources will come in handy for specific Git subjects. interactive mode or on its own with `git bisect run` to find the problematic code commit that needs to be fixed. -* [How Microsoft uses Git](https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/use-git-microsoft) - gives a high-level overview of their repository structure and hosting - at the extremely large scale organization. - * [GitTips](https://git.wiki.kernel.org/index.php/GitTips) is a list of pro tips to clean up common issues and how to dive through Git history to find specific text. @@ -223,6 +219,11 @@ workflow. These resources will come in handy for specific Git subjects. system. This is an awesome read to get a view on how Git works under the commands you're using to manipulate these objects. +* [How to Undo Mistakes With Git Using the Command Line](https://www.youtube.com/watch?v=lX9hsdsAeTk) + is a video that covers topics like resetting a file to an old revision, + recovering deleted commits, squashing multiple commits into one with + interactive rebase and recovering deleted branches. + ## Git Workflows Teams of developers can use Git in varying workflows because of Git's @@ -239,10 +240,6 @@ minimize merge conflicts. why at GitHub they do not use the git-flow model and provides an alternative that solves some of the issues they found with git-flow. -* [Git Workflows That Work](http://blog.endpoint.com/2014/05/git-workflows-that-work.html) - is a helpful post with diagrams to show how teams can create a Git workflow - that will help their development process. - * [Comparing workflows](https://www.atlassian.com/git/tutorials/comparing-workflows) provides a slew of examples for how developers on a team can handle merge conflicts and other situations that commonly arise when using Git. diff --git a/content/pages/03-data/00-data.markdown b/content/pages/03-data/00-data.markdown index a2c996e10..39a767500 100644 --- a/content/pages/03-data/00-data.markdown +++ b/content/pages/03-data/00-data.markdown @@ -12,7 +12,7 @@ subsections, including (in no particular order): * data processing / wrangling * machine learning -* data analysis +* [data analysis](/data-analysis.html) * [visualization](/data-visualization.html) * geospatial mapping * persistence via [relational databases](/databases.html) and @@ -154,6 +154,17 @@ sets. have evolved over the past 20ish years based on his first-hand experience as a leader and member in that community. +* The State of [Python Speech Recognition](https://www.assemblyai.com/blog/the-state-of-python-speech-recognition-in-2021) + in 2021 is a practical overview of a specific area in data: extracting text + from voice recording data. Looking at verticals like this one can make it + easier to understand changes that are occurring in some parts of data and + programming that could be applied to other areas. + +* [Automated Data Wrangling](https://catalyst.coop/2021/05/23/automated-data-wrangling/) + covers cleaning, labeling, and automating the bunch of activities + that are typically necessary before analysis and data usage can + begin for a project. + * The [Open Source Data Science Masters](http://datasciencemasters.org/) is a well-crafted free curriculum and set of resources for students who want to learn both the theory and technologies for working with data. diff --git a/content/pages/03-data/01-databases.markdown b/content/pages/03-data/01-databases.markdown index 5eed45185..bad071a2c 100644 --- a/content/pages/03-data/01-databases.markdown +++ b/content/pages/03-data/01-databases.markdown @@ -100,7 +100,7 @@ Find out about Python applications with a MySQL backed on the dedicated To work with a relational database using Python, you need to use a code library. The most common libraries for relational databases are: -* [psycopg2](http://initd.org/psycopg/) +* [psycopg](https://www.psycopg.org/) ([source code](https://github.com/psycopg/psycopg2)) for PostgreSQL. @@ -112,8 +112,6 @@ library. The most common libraries for relational databases are: * [cx\_Oracle](https://oracle.github.io/python-cx_Oracle/index.html) for Oracle Database ([source code](https://github.com/oracle/python-cx_Oracle)). - Oracle moved their - [open source driver code from SourceForge to GitHub in 2017](https://blogs.oracle.com/developers/oracle-database-python-driver-now-on-github). SQLite support is built into Python 2.7+ and therefore a separate library @@ -183,7 +181,7 @@ speed on SQL if you have never previously used it. elaborates on one of the trickiest parts of writing SQL statements that bridge one or more tables: the `JOIN`. -* [Writing better SQL](http://www.craigkerstiens.com/2016/01/08/writing-better-sql/) +* [Writing more legible SQL](https://www.craigkerstiens.com/2016/01/08/writing-more-legible-sql/) is a short code styling guide to make your queries easier to read. * [SQL Intermediate](https://www.dataquest.io/blog/sql-intermediate/) is a @@ -265,7 +263,7 @@ speed on SQL if you have never previously used it. 1. Install PostgreSQL on your server. Assuming you went with Ubuntu run ``sudo apt-get install postgresql``. -1. Make sure the [psycopg2](http://initd.org/psycopg/) library is in your +1. Make sure the [psycopg](https://www.psycopg.org/) library is in your application's dependencies. 1. Configure your web application to connect to the PostgreSQL instance. diff --git a/content/pages/03-data/02-postgresql.markdown b/content/pages/03-data/02-postgresql.markdown index 26954a7f8..5e700895a 100644 --- a/content/pages/03-data/02-postgresql.markdown +++ b/content/pages/03-data/02-postgresql.markdown @@ -57,7 +57,7 @@ architecture. To work with relational databases in Python you need to use a database driver, which is also referred to as a database connector. The most common driver library for working with PostgreSQL is -[psycopg2](http://initd.org/psycopg/). There is +[psycopg](https://www.psycopg.org/). There is [a list of all drivers on the PostgreSQL wiki](https://wiki.postgresql.org/wiki/Python), including several libraries that are no longer maintained. If you're working with the @@ -126,13 +126,7 @@ walkthroughs I've read. * This article explains how and why PostgreSQL can handle [full text searching](http://blog.lostpropertyhq.com/postgres-full-text-search-is-good-enough/) - for many use cases. If you're going down this route, read - [this blog post that explains how one developer implemented PostgreSQL full text search with SQLAlchemy](http://blog.garage-coding.com/2015/12/18/postgres-fulltext-search.html). - -* [django-postgres-copy](http://django-postgres-copy.californiacivicdata.org/en/latest/) - is a tool for bulk loading data into a PostgreSQL database based on Django models. - [Say hello to our new open-source software for loading bulk data into PostgreSQL](http://www.californiacivicdata.org/2015/07/17/hello-django-postgres-copy/) - is an introduction to using the tool in your own projects. + for many use cases. * [How to speed up tests in Django and PostgreSQL](http://nemesisdesign.net/blog/coding/how-to-speed-up-tests-django-postgresql/) explains some hacks for making your schema migration-backed run quicker. diff --git a/content/pages/03-data/04-sqlite.markdown b/content/pages/03-data/04-sqlite.markdown index 2cb4cef26..5c7f3fd00 100644 --- a/content/pages/03-data/04-sqlite.markdown +++ b/content/pages/03-data/04-sqlite.markdown @@ -86,6 +86,10 @@ tutorials will help you get started. It's a great short read which shows that the code is well-tested and maintained. +* [SQLite is not a toy database](https://antonz.org/sqlite-is-not-a-toy-database/) + is a whirlwind overview of some of the best aspects of SQLite and why + you should use it. + * [Data Analysis of 8.2 Million Rows with Python and SQLite](https://plot.ly/ipython-notebooks/big-data-analytics-with-pandas-and-sqlite/) explains how you can load a large dataset in to SQLite and visualize it using the Plotly service. @@ -139,6 +143,9 @@ you are having with SQLite rather than going through a general tutorial. digs into the internals of SQLite and shows some bugs found (and since fixed) while the author was researching the SQLite source code. +* [How to Store Multimedia Files in a SQLite3 Database with Python](https://www.twilio.com/blog/intro-multimedia-file-upload-python-sqlite3-database) + goes through the Python code for storing and accessing BLOB-type objects. + * [Going Fast with SQLite and Python](http://charlesleifer.com/blog/going-fast-with-sqlite-and-python/) shares essential knowledge for working effectively with SQLite in Python, particularly when it comes to transactions, concurrency and commits. @@ -148,11 +155,6 @@ you are having with SQLite rather than going through a general tutorial. [object-relational mapper (ORM)](/object-relational-mappers-orms.html) to implement virtual tables and aggregates on top of SQLite. -* [Use SQLite with Django On AWS Lambda with Zappa](https://blog.zappa.io/posts/use-sqlite-with-django-on-aws-lambda-with-zappa) - provides an example `dev_settings.py` file for - locally testing a [Django](/django.html) application intended for - [AWS Lambda](/aws-lambda.html). - * [SQLite Database Authorization and Access Control with Python](http://charlesleifer.com/blog/sqlite-database-authorization-and-access-control-with-python/) covers how to control access to the SQLite database connection and file even though SQLite normally allows unauthorized access by design. diff --git a/content/pages/03-data/05-object-relational-mappers.markdown b/content/pages/03-data/05-object-relational-mappers.markdown index df5ce8bb0..6451908e8 100644 --- a/content/pages/03-data/05-object-relational-mappers.markdown +++ b/content/pages/03-data/05-object-relational-mappers.markdown @@ -64,7 +64,7 @@ there was a pressing reason. Python ORM libraries are not required for accessing relational databases. In fact, the low-level access is typically provided by another library called a *database connector*, such as -[psycopg](http://initd.org/psycopg/) (for PostgreSQL) +[psycopg](https://www.psycopg.org/) (for PostgreSQL) or [MySQL-python](https://pypi.org/project/MySQL-python/1.2.5) (for MySQL). Take a look at the table below which shows how ORMs can work with different web frameworks and connectors and relational databases. diff --git a/content/pages/03-data/08-django-orm.markdown b/content/pages/03-data/08-django-orm.markdown index e8c08aaee..59bb49f8b 100644 --- a/content/pages/03-data/08-django-orm.markdown +++ b/content/pages/03-data/08-django-orm.markdown @@ -192,3 +192,6 @@ following resources should get you past the initial hurdles. fields with a checking constraint and a web form that ensures all of the fields sum up to a precise amount, such as 100%. +* [Learn Django ORM - Query and Filters](https://www.youtube.com/playlist?list=PLOLrQ9Pn6cazjoDEnwzcdWWf4SNS0QZml) + is a video tutorials series that gives an overview of the ORM's + querying and filtering capabilities. diff --git a/content/pages/03-data/10-no-sql-datastore.markdown b/content/pages/03-data/10-no-sql-datastore.markdown index 05e47b705..c6c9b4683 100644 --- a/content/pages/03-data/10-no-sql-datastore.markdown +++ b/content/pages/03-data/10-no-sql-datastore.markdown @@ -189,10 +189,6 @@ representing a person could have a property of "female" or "male". * [CAP Theorem overview](http://natishalom.typepad.com/nati_shaloms_blog/2010/10/nocap.html) presents the basic constraints all databases must trade off in operation. -* This post on [What is a NoSQL database? Learn By Writing One in Python](http://jeffknupp.com/blog/2014/09/01/what-is-a-nosql-database-learn-by-writing-one-in-python/) - is a detailed article that breaks the mystique behind what some forms - of NoSQL databases are doing under the covers. - * The [CAP Theorem series](http://blog.thislongrun.com/2015/03/the-cap-theorem-series.html) explains concepts related to NoSQL such as what is ACID compared to CAP, CP versus CA and high availability in large scale deployments. diff --git a/content/pages/03-data/11-redis.markdown b/content/pages/03-data/11-redis.markdown index c05d04741..3600ddae2 100644 --- a/content/pages/03-data/11-redis.markdown +++ b/content/pages/03-data/11-redis.markdown @@ -84,10 +84,6 @@ Redis should be customized out of its default configuration to secure it against unauthorized and unauthenticated users. These resources provide some advice on Reids security and guarding against data breaches. -* [Pentesting Redis servers](http://averagesecurityguy.info/2015/09/17/pentesting-redis-servers/) - shows that security is important not only on your application but also - the databases you're using as well. - * Redis, just as with any relational or NoSQL database, needs to be secured based on [security guidelines](http://www.antirez.com/news/96). There is also a post where the main author of Redis diff --git a/content/pages/03-data/12-mongodb.markdown b/content/pages/03-data/12-mongodb.markdown index 94826bfd6..665144fec 100644 --- a/content/pages/03-data/12-mongodb.markdown +++ b/content/pages/03-data/12-mongodb.markdown @@ -67,6 +67,12 @@ command line and query language. [Google Cloud Function](/google-cloud-functions.html) to store persistent data while running on a serverless platform. +* [Everything You Know About MongoDB is Wrong!](https://developer.mongodb.com/article/everything-you-know-is-wrong) + lists many of the common thoughts developers have about MongoDB and + why some of them are misconceptions. This is a good read for + developers who used MongoDB several years ago and want to know what + major improvements have been made since then. + ### MongoDB security NoSQL databases can be a weak spot in a production deployment environment, @@ -93,11 +99,6 @@ security controls so make sure to lock down your instances. installing and using MongoDB on your own instance. The post covers authentication, SSL and firewalls. -* [Securing MongoDB using Let's Encrypt certificate](https://zohaib.me/securing-mongodb-using-lets-encrypt/) - gives a configuration that encrypts that traffic coming from and - going to your MongoDB instances using free - [Let's Encrypt certificates](https://letsencrypt.org/). - * This 4 post securing MongoDB series covers [Data Security Requirements for Regulatory Compliance](https://www.mongodb.com/blog/post/securing-mongodb-part-1-data-security-requirements-for-regulatory-compliance), [Database Access Control](https://www.mongodb.com/blog/post/securing-mongodb-part-2-database-access-control), @@ -105,11 +106,6 @@ security controls so make sure to lock down your instances. and [Environmental Control & Database Management](https://www.mongodb.com/blog/post/securing-mongodb-part-4-environmental-control-and-database-management). -* Lightweight Directory Access Protocol (LDAP) is common in many - established company environments for security. This post on - [How to Configure LDAP Authentication for MongoDB](https://www.mongodb.com/blog/post/how-to-configure-LDAP-authentication-for-mongodb) - goes over how to authenticate users via LDAP who are using MongoDB. - ### Python with MongoDB resources MongoDB is straightforward to use in a Python application when a driver diff --git a/content/pages/03-data/16-pandas.markdown b/content/pages/03-data/16-pandas.markdown index 1ac1e2b5b..a0750eaae 100644 --- a/content/pages/03-data/16-pandas.markdown +++ b/content/pages/03-data/16-pandas.markdown @@ -70,15 +70,18 @@ is a data structures and analysis library. and charts that show the pay off period broken down by interest and principal. +* [Efficiently cleaning text with pandas](https://pbpython.com/text-cleaning.html) + provides a really great practical tutorial on different approaches + for cleaning a large data set so that you can begin to do your analysis. + The tutorial also shows how to use the + [sidetable](https://github.com/chris1610/sidetable) library, which + creates summary tables of a DataFrame. + * [tabula-py: Extract table from PDF into Python DataFrame](https://blog.chezo.uno/tabula-py-extract-table-from-pdf-into-python-dataframe-6c7acfa5f302) presents how to use the Python wrapper for the [Tabula](https://tabula.technology/) library that makes it easier to extract table data from PDF files. -* [Analyzing Browser History Using Python and Pandas](https://applecrazy.github.io/blog/2017-11-12/analyzing-browser-hist-using-python) - shows how to take data from Google Chrome and start to visualize it - with pandas and [matplotlib](/matplotlib.html). - * [Time Series Forecast Case Study with Python: Monthly Armed Robberies in Boston](https://machinelearningmastery.com/time-series-forecast-case-study-python-monthly-armed-robberies-boston/) walks through the data wrangling, analysis and visualization steps with a public data set of murders in Boston from 1966 to 1975. This @@ -129,3 +132,21 @@ is a data structures and analysis library. * [How to convert JSON to Excel with Python and pandas](https://www.marsja.se/how-to-convert-json-to-excel-python-pandas/) provides instructions for creating a spreadsheet out of JSON file. + +* [Loading large datasets in Pandas](https://towardsdatascience.com/loading-large-datasets-in-pandas-11bdddd36f7b) + explains how to get around the `MemoryError` issue that occurs + when using `read_csv` because the data set is larger than the + available memory on a machine. You can use chunking with + the `read_csv` function to divide the data set into smaller parts that + each can be loaded into memory. Alternatively, you can use a + [SQLite database](/sqlite.html) to create a [relational database](/databases.html) + with the data then use SQL queries or an + [object-relational mapper (ORM)](/object-relational-mappers-orms.html) + to load the data and perform analysis in pandas. + +* Real-world Excel spreadsheets are often a mess of unstructured data, so + this tutorial on + [Reading Poorly Structured Excel Files with Pandas](https://pbpython.com/pandas-excel-range.html) + gives example code for extracting only part of a file as well + as reading ranges and tables. + diff --git a/content/pages/03-data/17-scipy-numpy.markdown b/content/pages/03-data/17-scipy-numpy.markdown index b6350d46a..00763c2d0 100644 --- a/content/pages/03-data/17-scipy-numpy.markdown +++ b/content/pages/03-data/17-scipy-numpy.markdown @@ -44,11 +44,6 @@ following resources are broader walkthroughs for the SciPy ecosystem: plotter. This is a very cool example project that ties together the scientific world and the art world. -* [Lectures in Quantitative Economics: SciPy](https://lectures.quantecon.org/py/scipy.html) - provides a good overview of SciPy compared to the specific NumPy - project, as well as explanations for the wrappers SciPy provides - over lower-level FORTRAN libraries. - * [A plea for stability in the SciPy ecosystem](http://blog.khinsen.net/posts/2017/11/16/a-plea-for-stability-in-the-scipy-ecosystem/) presents concerns from one scientist's perspective about how fast the Python programming ecosystem changes and that code can become backwards @@ -67,12 +62,19 @@ following resources are broader walkthroughs for the SciPy ecosystem: computation and how to perform operations that get the results you need for your data analysis. +* [Scientific Computing in Python: Introduction to NumPy and Matplotlib](https://sebastianraschka.com/blog/2020/numpy-intro.html) + is a detailed tutorial that goes through the basics for NumPy and + then connects it to [Matplotlib](/matplotlib.html). + * [Math to Code](https://mathtocode.com/) provides an interactive tutorial to learn how to implement math in NumPy. * [101 NumPy Exercises for Data Analysis](https://www.machinelearningplus.com/python/101-numpy-exercises-python/) + has a bunch of questions and answers to common ways to work with NumPy + and is useful to understand what you can do with this library. * [NumPy: creating and manipulating numerical data](http://www.scipy-lectures.org/intro/numpy/index.html) + contains many code examples for common operations. * [Python NumPy Array Tutorial](https://www.datacamp.com/community/tutorials/python-numpy-tutorial) is a starter tutorial specifically focused on using and working diff --git a/content/pages/03-data/18-data-visualization.markdown b/content/pages/03-data/18-data-visualization.markdown index af18aada8..e56c2b2dc 100644 --- a/content/pages/03-data/18-data-visualization.markdown +++ b/content/pages/03-data/18-data-visualization.markdown @@ -91,11 +91,6 @@ Sometimes you need inspiration from other sources to figure out what you want to build. The following links have made me excited about data visualization and gave me ideas for what to build. -* [Roads to Rome](http://roadstorome.moovellab.com/) is a beautiful - visualization showing the data behind the expression "all roads lead - to Rome" and whether or not there is a "Rome" central city in every - country. - * [Monarchs](https://thebackend.dev/monarchs/) is a wonderful 1,000 year history visual of European rulers. The developer also wrote an in-depth article on @@ -107,10 +102,6 @@ visualization and gave me ideas for what to build. and dark sides, the main characters, various bits about the Force and other data extracted from the movies. -* [Big League Graphs](https://bigleaguegraphs.com/) presents a bunch of - creative ways to view data for sports such as basketball, baseball and - hockey. - * [What do numbers look like?](https://johnhw.github.io/umap_primes/index.md.html) is a Python 3 dimensional visualization of millions of integers, colored by special factors such as prime and Fibonacci numbers. diff --git a/content/pages/03-data/19-bokeh.markdown b/content/pages/03-data/19-bokeh.markdown index 668538d90..22f8d55cc 100644 --- a/content/pages/03-data/19-bokeh.markdown +++ b/content/pages/03-data/19-bokeh.markdown @@ -61,12 +61,6 @@ basic syntax will change as the library's API is not yet stable. an appropriate format then explains the code that uses Bokeh to visualize it. -* [Data is beautiful: Visualizing Roman imperial dynasties](http://machineloveus.com/data-is-beautiful-visualizing-roman-imperial-dynasties/) - provides a walkthrough for creating a gorgeous visualization based on - historical Roman data. The post is about more than just the visual, it also - goes into the ideation, data wrangling and analysis phases that came - before using Bokeh to show the results. - * [Visualizing with Bokeh](https://programminghistorian.org/en/lessons/visualizing-with-bokeh) gives a detailed explanation with the code for number Bokeh visuals you can output while working with a [pandas](/pandas.html) data set. @@ -105,13 +99,6 @@ basic syntax will change as the library's API is not yet stable. project has the code to create a simple chart with Bokeh and [Flask](/flask.html). -* [Bokeh vs Dash — Which is the Best Dashboard Framework for Python?](https://blog.sicara.com/bokeh-dash-best-dashboard-framework-python-shiny-alternative-c5b576375f7f) - contains a single project that was written in both Dash and Bokeh. The - author gives his subjective view on the implementation difficulty - although the web application only contained a single type of data - visualization so it is hard to drawn any real conclusions from his - opinion. - * [Realtime Flight Tracking with Pandas and Bokeh](https://www.geodose.com/2019/01/realtime-flight-tracking-pandas-bokeh-python.html) provides a great example of combining [pandas](/pandas.html) for structuring data with Bokeh for visualization. diff --git a/content/pages/03-data/21-matplotlib.markdown b/content/pages/03-data/21-matplotlib.markdown index 14f4d2f64..9c60d3740 100644 --- a/content/pages/03-data/21-matplotlib.markdown +++ b/content/pages/03-data/21-matplotlib.markdown @@ -21,10 +21,6 @@ toolkits. is an awesome getting started tutorial that breaks through the confusing beginner steps so you can quick start using the plotting library. -* [Web Scraping With Python: Scrapy, SQL, Matplotlib To Gain Web Data Insights](http://www.scrapingauthority.com/python-scrapy-mysql-and-matplotlib-to-gain-web-data-insights/) - is a long and comprehensive tutorial that walks through obtaining, - cleaning and visualizing data. - * [Matplotlib Cheat Sheet: Plotting in Python](https://www.datacamp.com/community/blog/python-matplotlib-cheat-sheet) contains some handy snippets of code to perform common plotting operations in Matplotlib. diff --git a/content/pages/03-data/25-oracle.markdown b/content/pages/03-data/25-oracle.markdown new file mode 100644 index 000000000..bf6b3cdc0 --- /dev/null +++ b/content/pages/03-data/25-oracle.markdown @@ -0,0 +1,165 @@ +title: Oracle +category: page +slug: Oracle +sortorder: 0325 +toc: False +sidebartitle: Oracle +meta: Oracle Database is an enterprise relational database management system. + + +[Oracle Database](http://www.oracle.com/) is an enterprise +[relational database](/databases.html). It can run transaction processing, +data warehousing, and multi-model database workloads such as machine +learning, spatial, and graph analysis. Recent versions of Oracle Database +also added support for JSON and blockchain use cases, and the software +can be run in on-premise, cloud or hybrid environments. + +Oracle logo. + + +## How does Oracle fit with Python? +The Python community and Oracle have a long history. The excellent Python Database API-compliant "cx_Oracle" interface for Oracle Database was first created by the user community in 1998 and is now being enhanced and maintained by Oracle. The [cx_Oracle](https://oracle.github.io/python-cx_Oracle/) module also underpins the [Oracle Machine Learning for Python](https://www.youtube.com/watch?v=P861m__PEMQ) engine. Oracle's high-performance GraalVM framework supports an implementation of Python called [GraalPython](https://github.com/oracle/graalpython). + + +## Why is Oracle Database a great choice? +Oracle Database is cross-platform, supporting multiple hardware platforms and various operating systems. Developers and companies of all sizes rely on its proven industry-leading performance, scalability, reliability, and security. +As data volumes rise exponentially, new data types and data models are required to support modern applications. Oracle Database supports the following data types at no extra cost: + +* [JSON](https://docs.oracle.com/en/database/oracle/oracle-database/19/adjsn/index.html) +* [Blockchain](https://docs.oracle.com/en/database/oracle/oracle-database/21/nfcon/details-oracle-blockchain-table-282449857.html) +* [XML](https://www.oracle.com/database/technologies/appdev/xmldb.html) +* [Object](https://docs.oracle.com/database/121/ADOBJ/adobjint.htm#ADOBJ00101) +* [Graph](https://www.oracle.com/database/graph/) +* [Spatial](https://www.oracle.com/database/spatial/) +* [Time Series](https://docs.oracle.com/en/database/oracle/oracle-database/19/dmcon/time-series.html) +* Relational + +With support for scale-out database clusters, sharded distributed systems, and disaster recovery with continuous application availability, there is no shortage of features to guarantee the Database continues to run uninterrupted 24/7. + +Oracle makes its enterprise-class database readily available to developers with its free on-premises edition Oracle Database XE or on the Oracle public cloud with an Always Free Cloud account. In addition, Oracle Autonomous Database is a popular choice for developers as no database management or tuning is required, leaving developers to do what they do best – writing code for their applications. + + +## Connecting to Oracle Database with Python +As with any database, applications require a connector or driver to connect to the Oracle Database. The Python DB API-compliant [cx_Oracle](https://github.com/oracle/python-cx_Oracle) interface provides developers access to standard and advanced Oracle Database features, such as SQL execution and document storage APIs. It also gives users access to network traffic encryption capabilities and Oracle's leading high availability features. + +[Code examples](https://oracle.github.io/python-cx_Oracle/samples/tutorial/Python-and-Oracle-Database-Scripting-for-the-Future.html) and free workshops such as the introductory [Python and Oracle for Developers Workshop](https://apexapps.oracle.com/pls/apex/dbpm/r/livelabs/view-workshop?wid=766) and a full-stack development workshop using [Python with SQLAlchemy to Oracle Database](https://apexapps.oracle.com/pls/apex/dbpm/r/livelabs/view-workshop?wid=911&clear=180&session=16650643444916) are available. + +cx Oracle driver. + +You can use many Python frameworks and [object-relational mappers (ORMs)](/object-relational-mappers-orms.html) with Oracle Database. ORMs abstract the tables and objects in a relational database to objects that Python developers can manipulate and operate on. [SQLAlchemy](/sqlalchemy.html) and Django are popular ORMs. SQLAlchemy is used by Pandas, which is very popular with Oracle users. +The table below shows the relationship between web framework, ORM, driver, and the Oracle Database. + +Examples of how varying Python ORMs can work with Oracle and the cx Oracle connector. + +Learn more about +[Python ORMs on that dedicated topic page](/object-relational-mappers-orms.html). + +ORMs provide a familiar programming model for Python developers, but sometimes you want that extra performance and operate closer to SQL objects. Oracle cx_Oracle offers several [functions](https://oracle.github.io/python-cx_Oracle/samples/tutorial/Python-and-Oracle-Database-Scripting-for-the-Future.html#binding) to deliver that performance. These functions include fetching data, binding data, executing PL/SQL, operating on LOBs, JSON documents, message passing with Oracle Advanced Queuing, and more. + + +## Oracle and Data Safety +According to Gartner, Oracle has one of the [highest data safety ratings](https://www.gartner.com/reviews/market/cloud-database-management-systems/vendor/oracle/product/oracle-database) in the industry, with a wide range of features for data protection and high availability. These features include: + +* [Database encryption](https://www.oracle.com/database/technologies/security/advanced-security.html) + +* [Access control to rows](https://www.oracle.com/database/technologies/security/label-security.html) in a table + +* [Database vault](https://www.oracle.com/database/technologies/security/db-vault.html) to restrict privileges and access + +* [Data redaction, subsetting, and masking](https://www.oracle.com/database/technologies/security/data-masking-subsetting.html) + +* All in one data security service in the Oracle Cloud with [Data Safe](https://www.oracle.com/database/technologies/security/data-safe.html) + +* Oracle also provides free tools such as the [Database Assessment Tool (DBSAT)](https://www.oracle.com/database/technologies/security/dbsat.html) to help you identify and remedy potential vulnerabilities. + +Oracle also provides numerous data recovery features, including: + +* Backup capabilities with [RMAN](https://www.oracle.com/database/technologies/high-availability/rman.html) + +* Restore point features with [Database Flashback](https://www.oracle.com/database/technologies/high-availability/flashback.html) + +* [Application continuity](https://www.oracle.com/database/technologies/high-availability/app-continuity.html) in the event of database failover to a standby + +For an overview of Oracle’s security and high availability architecture, see the following white papers: + +* [Maximum Availability Architecture](https://www.oracle.com/a/tech/docs/maa-onpremises-overview.pdf) (MAA) + +* [Maximum Security Architecture](https://blogs.oracle.com/cloudsecurity/post/oracles-maximum-security-architecture-for-database-security) (MSA) + + +## Python Specific Oracle Database resources +Many quick starts, tutorials, and workshops exist specifically for Python developers using Oracle Database. Below are some of the best ones to start with. + + +###Getting Started +If you are looking for a fast way to get started with Python and Oracle Database, check out these two quick start tutorials. These tutorials walk you through installing and setting up the environment you need to connect Python to Oracle Database. + +* [Quick Start: Developing Python Applications for Oracle Database](https://www.oracle.com/database/technologies/appdev/python/quickstartpythononprem.html) + +* [Quick Start: Developing Python Applications for Oracle Autonomous Database](https://www.oracle.com/database/technologies/appdev/python/quickstartpythononprem.html) + +Once you have done one of these, then continue with the popular [Python and Oracle Database Tutorial: Scripting for the Future](https://oracle.github.io/python-cx_Oracle/samples/tutorial/Python-and-Oracle-Database-Scripting-for-the-Future.html) to dive deeper to master the Python cx_Oracle interface and see how to build great Oracle Database applications. + + +###Using Different Frameworks with Oracle +* [How to Run SQL Queries with Pandas](https://www.oracle.com/news/connect/run-sql-data-queries-with-pandas.html) is a good blog using Pandas for quick and easy data manipulation in Python. + +* [Using Oracle with Pandas in OCI Data Science Notebooks](https://docs.oracle.com/en-us/iaas/tools/ads-sdk/latest/user_guide/loading_data/efficient_use_of_oracle_rdbms_with_ads.html) dives deeper into using Pandas with large datasets in data science applications. + +* [Using SQLAlchemy with Oracle Database](https://docs.sqlalchemy.org/en/14/dialects/oracle.html) provides an excellent toolkit for Python developers using SQLAlchemy as their ORM. + +* [Using Django with Python and Oracle Database](https://www.oracle.com/webfolder/technetwork/tutorials/obe/db/oow10/python_django/python_django.htm) is a tutorial from Oracle and shows the Django Framework with Python to an Oracle Database. + +* [Connecting Pony ORM to the Database](https://docs.ponyorm.org/database.html) is a friendly guide on using Pony with databases. + +* [How to use Python Flask with Oracle Database](https://blogs.oracle.com/opal/post/how-to-use-python-flask-with-oracle-database). + +* [Part 1: Docker for Oracle Database Applications in Node.js and Python](https://blogs.oracle.com/opal/post/part-1-docker-for-oracle-database-applications-in-nodejs-and-python). + +* [Part 2: Docker for Oracle Database Applications in Node.js and Python](https://blogs.oracle.com/opal/post/part-2-docker-for-oracle-database-applications-in-nodejs-and-python). + +* [Faster JSON with Python cx_Oracle and Oracle Database 21’s new OSON storage format](https://blogs.oracle.com/opal/post/faster-json-with-python-cx_oracle-81-and-oracle-database-21s-new-oson-storage-format). + + +###Workshops +The following hands-on, free workshops provide step-by-step instructions and walkthroughs in a live environment. + +* [Use Python with Oracle Database 19c](https://apexapps.oracle.com/pls/apex/dbpm/r/livelabs/view-workshop?wid=635&clear=180&session=3484600041895) is an Oracle LiveLabs workshop that shows how to write Python code to connect to and read data from an Oracle Database, including JSON data. + +* [Python and Oracle for Developers](https://apexapps.oracle.com/pls/apex/dbpm/r/livelabs/workshop-attendee-2?p210_workshop_id=766&p210_type=2&session=3484600041895) is an Oracle LiveLabs workshop that explores the features of the Python cx_Oracle interface for Oracle Database, including efficient techniques for connection management and statement handling. + +* [Full Stack Development using Python and deployment via OKE](https://apexapps.oracle.com/pls/apex/dbpm/r/livelabs/view-workshop?wid=911&clear=180&session=3484600041895) is an Oracle LiveLabs workshop that explores how to build and deploy a simple cloud-native application using the most common frameworks and the Oracle Cloud Infrastructure services. + + +## Cloud Development with Oracle Database +The following resources are good starting points for those looking to build applications in the Oracle Cloud and deploy applications in Docker containers and Kubernetes. + +* [The Complete Guide To Getting Up And Running With Docker And Kubernetes On The Oracle Cloud](https://blogs.oracle.com/developers/post/the-complete-guide-to-getting-up-and-running-with-docker-and-kubernetes-on-the-oracle-cloud). + +* [Oracle Cloud Blog](https://www.oc-blog.com/) has lots of interesting information on different aspects of Oracle Cloud. + +For developers looking to focus on application development in the Oracle Cloud and not have to worry about managing the Oracle Database, the Autonomous Database is a good choice. All management, including patching and upgrades, scalability, and security, are entirely autonomous. The following resources offer you a glimpse of its capabilities. + +* [Julien Dontcheff’s Database Blog](https://juliandontcheff.wordpress.com/category/autonomous/) is a good collection of technical posts with the Autonomous Database. + +* [SQL Maria](https://sqlmaria.com/category/autonomous-database/) also has some excellent posts on all things Oracle Database including Autonomous. + +* [An Introduction to Autonomous Database](https://questoraclecommunity.org/learn/blogs/oracles-autonomous-database-an-introduction/) gives you a good overview. + +* [Autonomous Database for researchers](https://blogs.oracle.com/research/post/a-roadmap-of-oracle-autonomous-database-benefits-for-research) is a good blog with details on some autonomous features. + + +##General Oracle Database Resources +Here are some Oracle tutorials and resources not specific to Python that can help you take advantage of the Oracle Database features. + +* [Oracle Technical Architecture](https://www.oracle.com/webfolder/technetwork/tutorials/architecture-diagrams/18/technical-architecture/database-technical-architecture.html) is from Oracle and has nice visuals and short paragraphs on the architecture of the Oracle Database. + +* [Oracle Database Internals](https://databaseinternalmechanism.com/oracle-database-internals/) is an excellent post explaining the architecture of the Oracle Database. + +* This [Oracle Performance Tuning](https://blog.quest.com/oracle-performance-tuning-a-5-step-approach-to-optimized-performance/) blog has a 5-step approach to tuning Oracle. + +* [Oracle RAC](https://databaseinternalmechanism.com/oracle-rac/) is another good post on the concepts of RAC, Oracle’s Real Application Cluster software for database high availability. + +* The [Oracle Database Security](https://www.oracle.com/database/technologies/security.html) web page has lots of information on Oracle’s solutions for security called “defense in depth.” + +* This is a good post on the [Top 5 Reasons to choose Oracle](https://www.dbta.com/Editorial/News-Flashes/Top-5-Reasons-to-Use-an-Oracle-Database-144191.aspx) for a production database. diff --git a/content/pages/04-web-development/00-web-development.markdown b/content/pages/04-web-development/00-web-development.markdown index 29b3fb98b..919cc6c94 100644 --- a/content/pages/04-web-development/00-web-development.markdown +++ b/content/pages/04-web-development/00-web-development.markdown @@ -71,12 +71,6 @@ world. [data bases](/databases.html), [task queues](/task-queues.html), [caching](/caching.html) and several other critical concepts. -* The [Evolution of the Web](http://www.evolutionoftheweb.com/) visualizes - how web browsers and related technologies have changed over time as well as - the overall growth of the Internet in the amount of data transferred. Note - that the visualization unfortunately stops around the beginning of 2013 but - it's a good way to explore what happened in the first 24 years. - * [What happens when?](https://github.com/alex/what-happens-when) is an incredibly detailed answer to the questions "What happens when you type google.com into your browser's address box and press enter?" that @@ -92,19 +86,6 @@ world. the creation of the URL. This is a great read that provides historical context for why things are the way they are with the web. -* [Web app checklist](http://dhilipsiva.com/webapp-checklist/) presents - good practices that developers building and [deploying](/deployment.html) - web applications should follow. Don't worry about having every single - one of these recommendations implemented before getting your site - live, but it is worthwhile to review the list to make sure there is not - something obvious you can handle in a few minutes that will improve - your site's security, performance or usability. - -* [Web application development is different and better](http://radar.oreilly.com/2014/01/web-application-development-is-different-and-better.html) - provides some context for how web development has evolved from writing - static HTML files into the complex JavaScript client-side applications - produced today. - * [The Browser Hacker's Guide to Instantly Loading Everything](https://www.youtube.com/watch?v=7vUs5yOuv-o) is a spectacular technical talk given by Addy Osmani at JSConf EU 2017 that has great bits of developer knowledge for both beginner and diff --git a/content/pages/04-web-development/01-web-frameworks.markdown b/content/pages/04-web-development/01-web-frameworks.markdown index 6b52e3680..2adde9c83 100644 --- a/content/pages/04-web-development/01-web-frameworks.markdown +++ b/content/pages/04-web-development/01-web-frameworks.markdown @@ -90,15 +90,15 @@ but it'll make most developers' lives easier in many cases. ### Comparing web frameworks +[Talk Python to Me had a podcast episode](https://talkpython.fm/episodes/show/149/4-python-web-frameworks-compared) +with a detailed comparison of the [Django](/django.html), +[Flask](/flask.html), Tornado and [Pyramid](/pyramid.html) frameworks. + Are you curious about how the code in a Django project is structured compared with Flask? Check out [this Django web application tutorial](https://www.twilio.com/docs/sms/tutorials/appointment-reminders-python-django) and then view [the same application built with Flask](https://www.twilio.com/docs/sms/tutorials/appointment-reminders-python-flask). -[Talk Python to Me had a podcast episode](https://talkpython.fm/episodes/show/149/4-python-web-frameworks-compared) -with a detailed comparison of the [Django](/django.html), -[Flask](/flask.html), Tornado and [Pyramid](/pyramid.html) frameworks. - There is also a repository called [compare-python-web-frameworks](https://github.com/mattmakai/compare-python-web-frameworks) where the same web application is being coded with varying Python web @@ -112,6 +112,18 @@ frameworks, templating engines and and the many other pieces that combine to make web frameworks useful to web developers. +* [12 requests per second](https://suade.org/dev/12-requests-per-second-with-python/) + examines how the traditionally synchronous web framework + [Flask](/flask.html) compares to an async framework like + [Sanic](/sanic.html) in an artificial, simple benchmark. The + results make it look like Sanic is far faster than Flask, but + once you add even a basic amount of functionality to a + project, including [databasel](/databases.html) queries + and templating, the results even out. Miguel Grinberg + also has a great read with broader results in this + article asking readers to + [Ignore All Web Performance Benchmarks, Including This One](https://blog.miguelgrinberg.com/post/ignore-all-web-performance-benchmarks-including-this-one). + * When you are learning how to use one or more web frameworks it's helpful to have an idea of what the code under the covers is doing. This post on building a @@ -150,10 +162,6 @@ frameworks, templating engines and I agree although I've found sessions and database ORMs to be a helpful part of a framework when done well. -* "[What is a web framework?](http://www.jeffknupp.com/blog/2014/03/03/what-is-a-web-framework/)" - is an in-depth explanation of what web frameworks are and their relation - to web servers. - * [Django vs Flask vs Pyramid: Choosing a Python Web Framework](https://www.airpair.com/python/posts/django-flask-pyramid) contains background information and code comparisons for similar web applications built in these three big Python frameworks. diff --git a/content/pages/04-web-development/02-django.markdown b/content/pages/04-web-development/02-django.markdown index 06ee0deee..3061d2eee 100644 --- a/content/pages/04-web-development/02-django.markdown +++ b/content/pages/04-web-development/02-django.markdown @@ -50,11 +50,6 @@ groups such as [Django District](http://www.meetup.com/django-district/), [San Francisco Django](http://www.meetup.com/The-San-Francisco-Django-Meetup-Group/) so new developers can get help when they are stuck. -There's some debate on whether -[learning Python by using Django is a bad idea](http://www.jeffknupp.com/blog/2012/12/11/learning-python-via-django-considered-harmful/). -However, that criticism is invalid if you take the time to learn the Python -syntax and language semantics first before diving into web development. - ## Django books and tutorials There are a slew of free or low cost resources out there for Django. Make @@ -126,7 +121,7 @@ These books and tutorials assume that you know the basics of building Django and want to go further to become much more knowledgeable about the framework. -* [2 Scoops of Django](https://www.twoscoopspress.com/collections/django/products/two-scoops-of-django-1-11) +* [2 Scoops of Django](https://www.feldroy.com/books/two-scoops-of-django-3-x) by Daniel Greenfeld and Audrey Roy is well worth the price of admission if you're serious about learning how to correctly develop Django websites. @@ -343,11 +338,6 @@ out how to build your own projects. This is a short list of some real-world example applications, and many more can be found on the [Django example projects and code](/django-code-examples.html) page. -* [Browser calls with Django and Twilio](https://www.twilio.com/docs/howto/walkthrough/browser-calls/python/django) - shows how to build a web app with Django and - [Twilio Client](https://www.twilio.com/client) to turn a user's web - browser into a full-fledged phone. Pretty awesome! - * [Openduty](https://github.com/ustream/openduty) is a website status checking and alert system similar to PagerDuty. @@ -364,6 +354,10 @@ real-world example applications, and many more can be found on the * [Taiga](https://github.com/taigaio/taiga-back) is a project management tool built with Django as the backend and AngularJS as the front end. +* [Chowist](https://github.com/huangsam/chowist) is a web application + that replicates core features of Yelp and adds a couple more bells + and whistles. + ## Open source code to learn Django There are many open source projects that rely on Django. diff --git a/content/pages/04-web-development/03-flask.markdown b/content/pages/04-web-development/03-flask.markdown index 18c708e54..fd7625f0e 100644 --- a/content/pages/04-web-development/03-flask.markdown +++ b/content/pages/04-web-development/03-flask.markdown @@ -205,9 +205,6 @@ combined with the example real-world projects listed in the next section. is a killer Flask tutorial with all the code needed to create a web app that can dial phones and receive inbound calls. -* Jeff Knupp provides some solid advice on how to - [productionize a Flask app](http://www.jeffknupp.com/blog/2014/01/29/productionizing-a-flask-application/). - * If you're looking for a fun tutorial with Flask and WebSockets, check out my blog post on creating [Choose Your Own Adventure Presentations with Reveal.js, Python and WebSockets](https://www.twilio.com/blog/2014/11/choose-your-own-adventure-presentations-with-reveal-js-python-and-websockets.html). @@ -298,7 +295,7 @@ about how to working on your codebase. * [Bean Counter](https://github.com/BouncyNudibranch/bean-counter) is an open source Flask app for tracking coffee. -* [FlaskBB](http://flaskbb.org/) is a Flask app for a discussion forum. +* [FlaskBB](https://flaskbb.org/) is a Flask app for a discussion forum. * [psdash](https://github.com/Jahaja/psdash) is an app built with Flask and psutils to display information about the computer it is running on. diff --git a/content/pages/04-web-development/04-bottle.markdown b/content/pages/04-web-development/04-bottle.markdown index 1f40db51c..01e543da4 100644 --- a/content/pages/04-web-development/04-bottle.markdown +++ b/content/pages/04-web-development/04-bottle.markdown @@ -29,7 +29,7 @@ Bottle is awesome for a few web development situations: Prototyping simple ideas is often easier with Bottle than a more opinionated web framework like [Django](/django.html) because Django projects start with a significant amount of boilerplate code. The -[Model-View-Template](https://docs.djangoproject.com/en/1.9/faq/general/#django-appears-to-be-a-mvc-framework-but-you-call-the-controller-the-view-and-the-view-the-template-how-come-you-don-t-use-the-standard-names) +[Model-View-Template](https://docs.djangoproject.com/en/stable/intro/tutorial03/) structure for Django apps within projects makes maintaining projects easier, but it can be cumbersome on starter projects where you're just playing with random ideas so you aren't worried about your diff --git a/content/pages/04-web-development/11-template-engines.markdown b/content/pages/04-web-development/11-template-engines.markdown index f8b7bac06..551a70bfb 100644 --- a/content/pages/04-web-development/11-template-engines.markdown +++ b/content/pages/04-web-development/11-template-engines.markdown @@ -161,6 +161,10 @@ know how they work to aid your debugging. The following resources examine existing template engine design as well as how to build your own engine when that's necessary for your projects. +* [Writing a Jinja-inspired template library in Python](https://notes.eatonphil.com/writing-a-template-library-in-python.html) + walks through how to create your own a simplified version of the + [Jinja](/jinja2.html) template engine as a learning exercise. + * [How a template engine works](https://fengsp.github.io/blog/2016/8/how-a-template-engine-works/) uses the template module in Tornado as an example to step through how a template engine produces output, from parsing the incoming string to diff --git a/content/pages/04-web-development/12-jinja2.markdown b/content/pages/04-web-development/12-jinja2.markdown index 23137b27d..75feb4fdc 100644 --- a/content/pages/04-web-development/12-jinja2.markdown +++ b/content/pages/04-web-development/12-jinja2.markdown @@ -8,17 +8,17 @@ meta: Jinja2 is a template engine written in Python for outputting formats such Jinja, also commonly referred to as -"[Jinja2](http://jinja.pocoo.org/docs/dev/)" to specify the newest +"[Jinja2](https://jinja.palletsprojects.com/en/3.0.x/templates/)" to specify the newest release version, is a Python [template engine](/template-engines.html) used to create HTML, XML or other markup formats that are returned to the user via an HTTP response. -Logo for the Jinja template engine project. +Logo for the Jinja template engine project. ## Why is Jinja2 useful? Jinja2 is useful because it has consistent template tag syntax and the project is cleanly extracted as -[an independent open source project](https://github.com/mitsuhiko/jinja2) so +[an independent open source project](https://github.com/pallets/jinja) so it can be used as a dependency by other code libraries.
Jinja2 is an implementation of the template engines concept. Learn more in the web development chapter or view the table of contents for all topics.
@@ -31,9 +31,8 @@ end a developer can code whatever she wants. ## Jinja2 origin and development The first recorded public released of Jinja2 was in -[2008 with 2.0rc1](http://jinja.pocoo.org/docs/dev/changelog/#version-2-0rc1). -Since then the engine has seen numerous updates and remains in active -development. +2008 with 2.0rc1. Since then the engine has seen numerous updates and +remains under active development. Jinja2 engine certainly wasn't the first template engine. In fact, Jinja2's syntax is inspired by Django's built-in template engine, which was released @@ -72,7 +71,7 @@ open source project author from having to reinvent a new templating style. the template engine. * The official - [Jinja2 template designer documentation](http://jinja.pocoo.org/docs/dev/templates/) + [Jinja2 templates documentation](https://flask.palletsprojects.com/en/2.0.x/tutorial/templates/) is exceptionally useful both as a reference as well as a full read-through to understand how to properly work with template tags. diff --git a/content/pages/04-web-development/15-web-design.markdown b/content/pages/04-web-development/15-web-design.markdown index b485938b3..58e9cc31f 100644 --- a/content/pages/04-web-development/15-web-design.markdown +++ b/content/pages/04-web-development/15-web-design.markdown @@ -83,6 +83,11 @@ this short list as my absolute favorites that help developers become design principles for building user experiences. Highly recommended even if just to see how the information is presented. +* [Building your color palette](https://refactoringui.com/previews/building-your-color-palette/) + explains why color pickers are not useful for most user interfaces + and how you should actually go about selecting your color palette + for a real world application. + * [How I Work with Color](https://medium.com/@JustinMezzell/how-i-work-with-color-8439c98ae5ed) is a fantastic article from a professional designer on how he thinks about color and uses it for certain effects in his designs. diff --git a/content/pages/04-web-development/17-css.markdown b/content/pages/04-web-development/17-css.markdown index f7d3658b2..d50968d09 100644 --- a/content/pages/04-web-development/17-css.markdown +++ b/content/pages/04-web-development/17-css.markdown @@ -119,6 +119,14 @@ web application's design. provides advice on how to write simpler, easier-to-maintain CSS code to reduce your need to rely on CSS preprocessors and build pipelines. +* [How to Detect Unused CSS or JavaScript](https://javascript.plainenglish.io/detect-unused-css-or-javascript-in-your-code-8d200ef07e50) + explains how to use [Chrome DevTools](https://developer.chrome.com/docs/devtools/) + to analyze a page's CSS and identify parts that are not used. Note that + even though a specific page does not use that CSS (or JS), there might + be another page that uses the same CSS files and *does* use that "unused" + code, so test your pages before and after making the changes to ensure + you did not inadvertently break something else! + * [CSS refresher notes](https://github.com/vasanthk/css-refresher-notes) is incredibly helpful if you've learned CSS in bits and pieces along the way and you now want to fill in the gaps in your knowledge. @@ -149,10 +157,6 @@ web application's design. * [Google's Web Fundamentals class](https://developers.google.com/web/fundamentals/) shows how to create responsive designs and performant websites. -* [Tailoring CSS for performance](http://programming.oreilly.com/2014/04/tailoring-css-for-performance.html) - is an interesting read since many developers do not consider the - implications of CSS complexity in browser rendering time. - * [Can I Use...](http://caniuse.com/) is a compatibility table that shows which versions of browsers implement specific CSS features. @@ -186,16 +190,6 @@ web application's design. talk about tables, because that was the only way to position anything back in the day. -* [30 seconds of CSS](https://30-seconds.github.io/30-seconds-of-css/) - provides short useful code snippets for you to learn from and use for - building your own web applications. - -* [CSS: The bad bits](https://www.joeforshaw.com/blog/css-the-bad-bits-and-how-to-avoid-them) - examines global scope, implicit percentage styling rules and the z-index - which can be difficult to use and require some restraint to ensure they - do not cause issues for the rest of your stylesheet rules as you create - and maintain your frontend. - * [Improving Your CSS with Parker](https://csswizardry.com/2016/06/improving-your-css-with-parker/) shows how to use the static CSS analysis tool [Parker](https://github.com/katiefenn/parker/) to improve your stylesheets. diff --git a/content/pages/04-web-development/24-react.markdown b/content/pages/04-web-development/24-react.markdown index 18ca6555b..518d5e929 100644 --- a/content/pages/04-web-development/24-react.markdown +++ b/content/pages/04-web-development/24-react.markdown @@ -35,6 +35,27 @@ tack on React to build your client-side user interfaces. quickly fall out of date while this one tends to stick to the basics that are relevant to beginners. +* [9 things every React.js beginner should know](https://camjackson.net/post/9-things-every-reactjs-beginner-should-know) + is not a tutorial but instead the author gives some strong opinions for + what beginners should know as they start learning React. + +* [React Bootstrap](https://react-bootstrap.github.io/) + ([source code](https://github.com/react-bootstrap/react-bootstrap) replaces + the existing Bootstrap JavaScript with React components that do not + rely on jQuery. + + +## Python+React tutorials +* [How to set up Django with React](https://mattsegal.dev/django-react.html) + presents one developer's opinionated way of combining a + [Django](/django.html)-powered back end with React on the front end, + including how to serve up static assets. + +* [Django REST with React (Django 2.0 and a sprinkle of testing)](https://www.valentinog.com/blog/tutorial-api-django-rest-react/) + combines a [Django](/django.html) plus + [Django REST Framework (DRF)](/django-rest-framework-drf.html) backend + with React on the front end and shows how to stich it all together. + * This Modern Django 4-part tutorial series is well-done, has [freely available source code](https://github.com/v1k45/ponynote) and includes: @@ -44,10 +65,6 @@ tack on React to build your client-side user interfaces. 3. [Creating an API and integrating with React](http://v1k45.com/blog/modern-django-part-3-creating-an-api-and-integrating-with-react/) 4. [Adding authentication to React SPA using DRF](http://v1k45.com/blog/modern-django-part-4-adding-authentication-to-react-spa-using-drf/) -* [Django REST with React (Django 2.0 and a sprinkle of testing)](https://www.valentinog.com/blog/tutorial-api-django-rest-react/) - combines a [Django](/django.html) plus - [Django REST Framework (DRF)](/django-rest-framework-drf.html) backend - with React on the front end and shows how to stich it all together. * [Build a Simple CRUD App with Python, Flask, and React](https://developer.okta.com/blog/2018/12/20/crud-app-with-python-flask-react) shows how to combine a [Flask](/flask.html) backend with React. @@ -56,15 +73,6 @@ tack on React to build your client-side user interfaces. is a Git repository with a code tutorial and instructions for how to follow along, as well as exercises to ensure you are tested as you go. -* [9 things every React.js beginner should know](https://camjackson.net/post/9-things-every-reactjs-beginner-should-know) - is not a tutorial but instead the author gives some strong opinions for - what beginners should know as they start learning React. - -* [React Bootstrap](https://react-bootstrap.github.io/) - ([source code](https://github.com/react-bootstrap/react-bootstrap) replaces - the existing Bootstrap JavaScript with React components that do not - rely on jQuery. - ### Other React resources * [React interview questions](https://tylermcginnis.com/react-interview-questions/) diff --git a/content/pages/04-web-development/25-vuejs.markdown b/content/pages/04-web-development/25-vuejs.markdown index 20a22a4d1..934f720b0 100644 --- a/content/pages/04-web-development/25-vuejs.markdown +++ b/content/pages/04-web-development/25-vuejs.markdown @@ -34,6 +34,11 @@ rich apps that run in web browsers. ### Vue.js resources +* [Building web apps with Vue and Django](https://dafoster.net/articles/2021/02/16/building-web-apps-with-vue-and-django-the-ultimate-guide/) + covers architectural decisions such as whether to use one or + two servers and then explains how to go down the one server + route with a [Django](/django.html) back end. + * [A friendly introduction to Vue.js](https://appendto.com/2016/11/a-friendly-introduction-to-vue-js/) contains the code and brief explanations of what it's doing so you can learn to create your first Vue app. diff --git a/content/pages/04-web-development/27-task-queues.markdown b/content/pages/04-web-development/27-task-queues.markdown index 073759f6e..70ec128c2 100644 --- a/content/pages/04-web-development/27-task-queues.markdown +++ b/content/pages/04-web-development/27-task-queues.markdown @@ -100,13 +100,6 @@ when scaling out a large deployment of distributed task queues. ## Open source examples that use task queues -* Take a look at the code in this open source - [Flask application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/flask) - and - [this Django application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/django) - for examples of how to use and deploy Celery with a Redis broker to - send text messages with these frameworks. - * [flask-celery-example](https://github.com/thrisp/flask-celery-example) is a simple Flask application with Celery as a task queue and Redis as the broker. @@ -128,10 +121,6 @@ when scaling out a large deployment of distributed task queues. is a detailed comparison of Amazon SQS, MongoDB, RabbitMQ, HornetQ and Kafka's designs and performance. -* [Queues.io](http://queues.io/) is a collection of task queue systems with - short summaries for each one. The task queues are not all compatible with - Python but ones that work with it are tagged with the "Python" keyword. - * [Why Task Queues](http://www.slideshare.net/bryanhelmig/task-queues-comorichweb-12962619) is a presentation for what task queues are and why they are needed. @@ -187,10 +176,6 @@ when scaling out a large deployment of distributed task queues. is a straightforward tutorial for setting up the Celery task queue for Django web applications using the Redis broker on the back end. -* [Three quick tips from two years with Celery](https://library.launchkit.io/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb) - provides some solid advice on retry delays, the -Ofair flag and global - task timeouts for Celery. - * [Asynchronous Tasks with Flask and Redis Queue](https://testdriven.io/asynchronous-tasks-with-flask-and-redis-queue) looks at how to configure Redis Queue to handle long-running tasks in a Flask app. diff --git a/content/pages/04-web-development/28-celery.markdown b/content/pages/04-web-development/28-celery.markdown index 5a614855c..6746997d1 100644 --- a/content/pages/04-web-development/28-celery.markdown +++ b/content/pages/04-web-development/28-celery.markdown @@ -7,11 +7,11 @@ sidebartitle: Celery meta: Celery is a task queue for executing work outside a Python web application HTTP request-response cycle. -[Celery](http://www.celeryproject.org/) is a [task queue](/task-queues.html) +[Celery](https://docs.celeryproject.org/) is a [task queue](/task-queues.html) implementation for [Python web applications](/web-development.html) used to asynchronously execute work outside the HTTP request-response cycle. -Celery task queue project logo. +Celery task queue project logo.
Celery is an implementation of the task queue concept. Learn more in the web development chapter or view the table of contents for all topics.
@@ -163,10 +163,6 @@ web framework of your choice. is a detailed walkthrough for setting up Celery with Django (although Celery can also be used without a problem with other frameworks). -* [Introducing Celery for Python+Django](http://www.linuxforu.com/2013/12/introducing-celery-pythondjango/) - provides an introduction to the Celery task queue with Django as the - intended framework for building a web application. - * [Asynchronous Tasks with Falcon and Celery](https://testdriven.io/asynchronous-tasks-with-falcon-and-celery) configures Celery with the [Falcon](/falcon.html) framework, which is less commonly-used in web tutorials. @@ -180,25 +176,14 @@ web framework of your choice. looks at how to configure Celery to handle long-running tasks in a Django app. + ### Celery deployment resources Celery and its broker run separately from your web and WSGI servers so it adds some additional complexity to your [deployments](/deployment.html). The following resources walk you through how to handle deployments and get the right configuration settings in place. -* The "Django in Production" series by - [Rob Golding](https://twitter.com/robgolding63) contains a post - specifically on [Background Tasks](http://www.robgolding.com/blog/2011/11/27/django-in-production-part-2---background-tasks/). - * [How to run celery as a daemon?](https://pythad.github.io/articles/2016-12/how-to-run-celery-as-a-daemon-in-production) is a short post with the minimal code for running the Celery daemon and Celerybeat as system services on Linux. -* [Celery in Production](http://www.caktusgroup.com/blog/2014/09/29/celery-production/) - on the Caktus Group blog contains good practices from their experience - using Celery with RabbitMQ, monitoring tools and other aspects not often - discussed in existing documentation. - -* [Three quick tips from two years with Celery](https://library.launchkit.io/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb) - provides some solid advice on retry delays, the `-Ofair` flag and global - task timeouts for Celery. diff --git a/content/pages/04-web-development/29-rq-redis-queue.markdown b/content/pages/04-web-development/29-rq-redis-queue.markdown index 55979bf3c..b9fe01a39 100644 --- a/content/pages/04-web-development/29-rq-redis-queue.markdown +++ b/content/pages/04-web-development/29-rq-redis-queue.markdown @@ -19,7 +19,7 @@ track of tasks in the queue that need to be executed. ### RQ resources * [Asynchronous Tasks in Python with Redis Queue](https://www.twilio.com/blog/asynchronous-tasks-in-python-with-redis-queue) is a quickstart-style tutorial that shows how to use RQ to fetch data - from the + from the [Mars Rover web API](https://data.nasa.gov/Space-Science/Mars-Rover-Photos-API/929k-jizu) and process URLs for each of the photos taken by NASA's Mars rover. There is also a follow-up post on @@ -30,6 +30,10 @@ track of tasks in the queue that need to be executed. * The [RQ intro post](http://nvie.com/posts/introducing-rq/) contains information on design decisions and how to use RQ in your projects. +* [Build a Ghostwriting App for Scary Halloween Stories with OpenAI's GPT-3 Engine and Task Queues in Python](https://www.twilio.com/blog/ghost-writer-spooky-task-queues-python-openai-gpt3) + is a fun tutorial that uses RQ with OpenAI's [GPT-3](/gpt-3.html) API + randomly write original stories inspired by creepy Halloween tales. + * [International Space Station notifications with Python and Redis Queue (RQ)](https://www.twilio.com/blog/2015/11/international-space-station-notifications-with-python-redis-queue-and-twilio-copilot.html) shows how to combine the RQ task queue library with Flask to send text message notifications every time a condition is met - in this blog diff --git a/content/pages/04-web-development/31-static-site-generator.markdown b/content/pages/04-web-development/31-static-site-generator.markdown index 7b07bc047..d7f3c0983 100644 --- a/content/pages/04-web-development/31-static-site-generator.markdown +++ b/content/pages/04-web-development/31-static-site-generator.markdown @@ -206,12 +206,6 @@ point a domain name to your site as well as provide HTTPS support. These guides walk through various ways of handling the static site deployment. -* [Static site hosting with S3 and Cloudflare](https://wsvincent.com/static-site-hosting-with-s3-and-cloudflare/) - shows how to set up an S3 bucket with Cloudflare in front as a CDN that - serves the content with HTTPS. You should be able to accomplish roughly - the same situation with Amazon Cloudfront, but as a Cloudflare user I - like their service for these static site configurations. - * Google Cloud provides a tutorial on how to use them to [host your static site](https://cloud.google.com/storage/docs/hosting-static-website). Note that you cannot currently use HTTPS on Google Storage servers, which is a @@ -221,7 +215,7 @@ deployment. making static site deployments and redeployments to Amazon Web Services easier. -* [Deploying a Static Blog with Continuous Integration](https://www.loxodrome.io/post/hugo-on-ci/) +* [Deploying a Static Blog with Continuous Integration](https://www.jameslmilner.com/post/hugo-on-ci/) uses a Hugo (a Golang-based static site generator) generated site as an example but the instructions can easily be used to deploy a Python-based static site generator output as well. diff --git a/content/pages/04-web-development/32-pelican.markdown b/content/pages/04-web-development/32-pelican.markdown index 064a3adf9..c1b0c0810 100644 --- a/content/pages/04-web-development/32-pelican.markdown +++ b/content/pages/04-web-development/32-pelican.markdown @@ -76,10 +76,6 @@ to hosting services such as Amazon S3 and GitHub Pages. take in new input markup formats, modify the generator process and add handy features such as a custom table of contents. -* [Pelican Sitemap and Pagination](http://www.vcheng.org/2014/02/22/pelican-sitemap-pagination/) - explains how to generate a `sitemap.xml` file for your static site that - includes all pages instead of just auto-included top-level pages. - * [Getting started with Pelican and GitHub pages](http://www.mattmakai.com/introduction-to-pelican.html) is a tutorial I wrote to use the Full Stack Python source code to create and deploy your first static site. diff --git a/content/pages/04-web-development/35-testing.markdown b/content/pages/04-web-development/35-testing.markdown index 1a5e2912e..7594935d2 100644 --- a/content/pages/04-web-development/35-testing.markdown +++ b/content/pages/04-web-development/35-testing.markdown @@ -176,10 +176,10 @@ use mocks in your test cases. provides a whole code example based on a blog project that shows how to use `mock` when testing. -* [Python Mocking 101: Fake It Before You Make It](https://blog.fugue.co/2016-02-11-python-mocking-101.html) +* [Python Mocking 101: Fake It Before You Make It](https://www.fugue.co/blog/2016-02-11-python-mocking-101) explains what mocking is and is not, and shows how to use the `patch` function to accomplish it in your project. - [Revisiting Unit Testing and Mocking in Python](https://blog.fugue.co/2017-07-18-revisiting-unit-testing-and-mocking-in-python.html) + [Revisiting Unit Testing and Mocking in Python](https://www.fugue.co/blog/2017-07-18-revisiting-unit-testing-and-mocking-in-python.html) is a follow-up post that expands upon using the `patch` function along with dependency injection. @@ -191,12 +191,6 @@ use mocks in your test cases. examines when mocks are necessary and when they are not as useful so you can avoid them in your test cases. -* [Mocking Redis & Expiration in Python](http://malexandre.fr/2017/10/08/mocking-redis--expiration-in-python/) - is a specific scenario where you would want to test your - [Redis](/redis.html)-dependent code but prefer to mock it rather than - ensure an installation and connection are present whenever you run - your tests. - * [Better tests for Redis integrations with redislite](https://www.obeythetestinggoat.com/better-tests-for-redis-integrations-with-redislite.html) is a great example of how using the right mocking library can clean up existing hacky testing code and make it more straightforward for diff --git a/content/pages/04-web-development/36-unit-testing.markdown b/content/pages/04-web-development/36-unit-testing.markdown index 86bebc128..9002c90d6 100644 --- a/content/pages/04-web-development/36-unit-testing.markdown +++ b/content/pages/04-web-development/36-unit-testing.markdown @@ -70,10 +70,6 @@ Python-specific applications. is a detailed tutorial for using the nose test runner for ensuring a Flask application is working properly. -* [Understanding unit testing](https://jeffknupp.com/blog/2013/12/09/improve-your-python-understanding-unit-testing/) - explains why testing is important and shows how to do it effectively in - your applications. - * [Unit testing with Python](http://www.drdobbs.com/testing/unit-testing-with-python/240165163) provides a high-level overview of testing and has diagrams to demonstrate what's going on in the testing cycle. diff --git a/content/pages/04-web-development/37-integration-testing.markdown b/content/pages/04-web-development/37-integration-testing.markdown index 72d844b82..8594b4492 100644 --- a/content/pages/04-web-development/37-integration-testing.markdown +++ b/content/pages/04-web-development/37-integration-testing.markdown @@ -36,9 +36,6 @@ during development so they can be addressed immediately. gives an example of a system that needs integration tests and shows how context managers can be used to address the problem. -* Pytest has a page on [integration good practices](http://doc.pytest.org/en/latest/goodpractices.html) - that you'll likely want to follow when testing your application. - * [Integration testing, or how to sleep well at night](http://enterprisecraftsmanship.com/2015/07/13/integration-testing-or-how-to-sleep-well-at-nights/) explains what integration tests are and gives an example. The example is coded in Java but still relevant when you're learning about integration diff --git a/content/pages/04-web-development/38-debugging.markdown b/content/pages/04-web-development/38-debugging.markdown index e2936c6ae..8cf576d27 100644 --- a/content/pages/04-web-development/38-debugging.markdown +++ b/content/pages/04-web-development/38-debugging.markdown @@ -137,10 +137,6 @@ give solid programming language-agnostic debugging advice. * [The art of debugging](https://remysharp.com/2015/10/14/the-art-of-debugging) provides a whirlwind overview for how to fix issues in your code. -* [How to debug JavaScript errors](https://rollbar.com/guides/how-to-debug-javascript/) - introduces some key debugging tools such as source maps that make - identifying errors on the client side much easier during development. - * [Linux debugging tools you'll love](https://jvns.ca/debugging-zine.pdf) is an awesome comic that covers the Linux ecosystem for debugging. diff --git a/content/pages/04-web-development/40-networking.markdown b/content/pages/04-web-development/40-networking.markdown index 5b3a98045..7e690cdba 100644 --- a/content/pages/04-web-development/40-networking.markdown +++ b/content/pages/04-web-development/40-networking.markdown @@ -13,6 +13,19 @@ web applications. ### Resources about networking +The "Let's code a TCP/IP stack" series along with its +[open source code](https://github.com/saminiir/level-ip) gives a ton of +context on how TCP/IP works while providing the code for implementing the +foundational pieces. You will likely need to pair this with a more theoretical +reference tutorial such as [RFC 1180](https://tools.ietf.org/html/rfc1180) +to have a more complete understanding of the protocol: + +1. [Ethernet & ARP](http://www.saminiir.com/lets-code-tcp-ip-stack-1-ethernet-arp/) +1. [IPv4 & ICMPv4](http://www.saminiir.com/lets-code-tcp-ip-stack-2-ipv4-icmpv4/) +1. [TCP Basics & Handshake](http://www.saminiir.com/lets-code-tcp-ip-stack-3-tcp-handshake/) +1. [TCP Data Flow & Socket API](http://www.saminiir.com/lets-code-tcp-ip-stack-4-tcp-data-flow-socket-api/) +1. [TCP Retransmission](http://www.saminiir.com/lets-code-tcp-ip-stack-5-tcp-retransmission/) + * [Monitoring and Tuning the Linux Networking Stack: Receiving Data](https://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/) along with [Monitoring and Tuning the Linux Networking Stack: Sending Data](https://blog.packagecloud.io/eng/2017/02/06/monitoring-tuning-linux-networking-stack-sending-data/) diff --git a/content/pages/04-web-development/41-https.markdown b/content/pages/04-web-development/41-https.markdown index ffe55e7c9..3bbb8239a 100644 --- a/content/pages/04-web-development/41-https.markdown +++ b/content/pages/04-web-development/41-https.markdown @@ -64,7 +64,3 @@ client web browser. cover the high-level information on the latest approved version of Transport Security Layer (TLS) 1.3. -* [How https works](https://www.sudhakar.online/programming/2015/08/09/https.html) - is a fun cartoon illustration that demonstrates the basic concepts of - a secure HTTP connection. - diff --git a/content/pages/04-web-development/42-websockets.markdown b/content/pages/04-web-development/42-websockets.markdown index fb8293c46..2a902f79c 100644 --- a/content/pages/04-web-development/42-websockets.markdown +++ b/content/pages/04-web-development/42-websockets.markdown @@ -7,10 +7,10 @@ sidebartitle: WebSockets meta: WebSockets are a protocol for full-duplex web communications. Learn about WebSockets on Full Stack Python. -A WebSocket is a [standard protocol](http://tools.ietf.org/html/rfc6455) for -two-way data transfer between a client and server. The WebSockets protocol -does not run over HTTP, instead it is a separate implementation on top of -[TCP](http://en.wikipedia.org/wiki/Transmission_Control_Protocol). +A WebSocket is a [standard protocol](https://datatracker.ietf.org/doc/html/rfc6455) +for two-way data transfer between a client and server. The WebSockets +protocol does not run over HTTP, instead it is a separate implementation +on top of [TCP](http://en.wikipedia.org/wiki/Transmission_Control_Protocol). ## Why use WebSockets? @@ -173,12 +173,6 @@ own project. that demos sending server generated events as well as input from users via a text box input on a form. -* The [realtime codenames game](https://github.com/joshporter1/codenames) - source code is a full-featured example for using WebSockets via - Flask-SocketIO. There is also a - [multi-part tutorial](https://secdevops.ai/weekend-project-part-1-creating-a-real-time-web-based-application-using-flask-vue-and-socket-b71c73f37df7) - that walks through the code. - * The [python-websockets-example](https://github.com/mattmakai/python-websockets-example) contains code to create a simple web application that provides WebSockets @@ -192,13 +186,6 @@ own project. Flask web app implementation that allows the audience to interact with WebSockets as I built out the application. -* [Creating a Real-time Web-based Application using Flask, Vue, and Socket.io: part 1](https://secdevops.ai/weekend-project-part-1-creating-a-real-time-web-based-application-using-flask-vue-and-socket-b71c73f37df7), - [part 2](https://secdevops.ai/weekend-project-part-2-turning-flask-into-a-real-time-websocket-server-using-flask-socketio-ab6b45f1d896) - and - [part 3](https://secdevops.ai/weekend-project-part-3-centralizing-state-management-with-vuex-5f4387ebc144) - are a complete front-to-backend WebSockets, Python and JavaScript front - end framework example with open source code. - * [Real-time in Python](http://mrjoes.github.io/2013/06/21/python-realtime.html) provides Python-specific context for how the server push updates were implemented in the past and how Python's tools have evolved to perform @@ -206,9 +193,7 @@ own project. * [websockets](https://github.com/aaugustin/websockets) is a WebSockets implementation for Python 3.3+ written with the - [asyncio](https://docs.python.org/3.4/library/asyncio.html) module (or with - [Tulip](https://code.google.com/p/tulip/) if you're working with - Python 3.3). + [asyncio](https://docs.python.org/3.9/library/asyncio.html) module. * [Speeding up Websockets 60X](https://www.willmcgugan.com/blog/tech/post/speeding-up-websockets-60x/) is a cool experiment in coding loops different ways to eek out more @@ -217,10 +202,6 @@ own project. of how tweaking and tuning can produce outsized returns in some applications. -* The [Choose Your Own Adventure Presentations](https://www.twilio.com/blog/2014/11/choose-your-own-adventure-presentations-with-reveal-js-python-and-websockets.html) - tutorial uses WebSockets via gevent on the server and socketio.js for - pushing vote count updates from the server to the client. - * [Adding Real Time to Django Applications](http://crossbar.io/docs/Adding-Real-Time-to-Django-Applications/) shows how to use Django and Crossbar.io to implement a publish/subscribe feature in the application. diff --git a/content/pages/04-web-development/43-webrtc.markdown b/content/pages/04-web-development/43-webrtc.markdown index 36cf18799..e34ac044a 100644 --- a/content/pages/04-web-development/43-webrtc.markdown +++ b/content/pages/04-web-development/43-webrtc.markdown @@ -26,11 +26,6 @@ browser) and server (usually a [web server](/web-servers.html)). dependencies. It allows 2 web browsers to exchange audio and video streams by using the `aiohttp` and `python-socketio` modules. -* [A real world guide to WebRTC](https://deepstreamhub.com/tutorials/protocols/webrtc-intro/) - goes through WebRTC fundamentals such as data channels, audio and video, - screen sharing and file transfers with the JavaScript code provided - for each concept. - * The [Introduction to WebRTC video series](https://www.youtube.com/watch?v=ujpIAWmK2Vo) ([part 2](https://www.youtube.com/watch?v=cw2iTgIW-uk) and @@ -38,10 +33,6 @@ browser) and server (usually a [web server](/web-servers.html)). at points but overall has a ton of good information that gives a solid overview of the technology. -* [Building a Snapchat-like app with WebRTC in the browser](https://tokbox.com/blog/building-a-snapchat-like-app-with-webrtc-in-the-browser/) - walks through the front end JavaScript for building a photo filter - application using the WebRTC browser APIs. - * [WebRTC issues and how to debug them](https://blog.codeship.com/webrtc-issues-and-how-to-debug-them/) explains the various ways that implementations can go wrong and where to start looking when you run into errors. diff --git a/content/pages/04-web-development/44-web-apis.markdown b/content/pages/04-web-development/44-web-apis.markdown index a0e1eea19..77ce68888 100644 --- a/content/pages/04-web-development/44-web-apis.markdown +++ b/content/pages/04-web-development/44-web-apis.markdown @@ -43,7 +43,7 @@ Webhooks are important because they enable two-way communication initiation for APIs. Webhook flexibility comes in from their definition by the API user instead of the API itself. -For example, in the [Twilio API](https://www.twilio.com/api) when a text +For example, in the [Twilio API](https://www.twilio.com/docs/api) when a text message is sent to a Twilio phone number Twilio sends an HTTP POST request webhook to the URL specified by the user. The URL is defined in a text box on the number's page on Twilio as shown below. @@ -91,11 +91,6 @@ on the number's page on Twilio as shown below. provides context for why JSON-based web services are more common today than SOAP which was popular in the early 2000s. -* [API tools for every occasion](https://medium.com/@orliesaurus/api-tools-for-every-occasion-10-api-tools-released-in-2015-i-can-t-live-without-d5947d9ca9c3) - provides a list of 10 tools that are really helpful when working with APIs - that are new in 2015. - - ## APIs learning checklist 1. Learn the API concepts of machine-to-machine communication with JSON and diff --git a/content/pages/04-web-development/45-microservices.markdown b/content/pages/04-web-development/45-microservices.markdown index bdf34a8b9..0b06889e6 100644 --- a/content/pages/04-web-development/45-microservices.markdown +++ b/content/pages/04-web-development/45-microservices.markdown @@ -40,22 +40,6 @@ ease further development and deployment. This approach is called the article is one of the best in-depth explanations for what microservices are and why to consider them as an architectural pattern. -* [Why microservices?](http://dev.otto.de/2016/03/20/why-microservices/) - presents some of the advantages, such as the dramatically increased number - of deployments per day, that a well-done microservices architecture can - provide in the right situation. Many organizational environments won't - allow this level of flexibility but if yours is one that will, it's worth - considering these points. - -* [On monoliths and microservices](http://dev.otto.de/2015/09/30/on-monoliths-and-microservices/) - provides some advice on using microservices in a fairly early stage of - a software project's lifecycle. - -* [Why Microservices?](https://dev.otto.de/2016/03/20/why-microservices/) - presents advantages microservices can bring to an existing monolithic - application where it is clear what needs to be broken down into smaller - components to make it easier to iterate and maintain. - * [Developing a RESTful microservice in Python](http://www.skybert.net/python/developing-a-restful-micro-service-in-python/) is a good story of how an aging Java project was replaced with a microservice built with Python and Flask. diff --git a/content/pages/04-web-development/46-webhooks.markdown b/content/pages/04-web-development/46-webhooks.markdown index 06e3b1984..6583c526d 100644 --- a/content/pages/04-web-development/46-webhooks.markdown +++ b/content/pages/04-web-development/46-webhooks.markdown @@ -23,19 +23,22 @@ otherwise independent web applications. ### Webhook resources +* [Building Webhooks Into Your Application: Guidelines and Best Practices](https://workos.com/blog/building-webhooks-into-your-application-guidelines-and-best-practices) + is an extensive high-level guide that defines what webhooks are, why you + will want to build them if you need to proactively notify other applications + of events, and what security considerations you need to have when + creating them. + * [What's a webhook?](https://sendgrid.com/blog/whats-webhook/) is a high-level explanation of this concept that also contains some basic security considerations when using them. -* [How to Listen for Webhooks with Python](https://blog.bearer.sh/consume-webhooks-with-python/) - has code examples in both [Flask](/flask.html) and [Django](/django.html) - for how to receive an HTTP POST webhook request, as well as how to test - it locally with Ngrok. +* [Webhooks for Beginners - Full Course](https://www.youtube.com/watch?v=41NOoEz3Tzc) + is an entire free video course that shows both how to use and implement + webhooks into applications. * [Should you build a webhooks API?](https://brandur.org/webhooks) -* [Webhooks do’s and dont’s: what we learned after integrating +100 APIs](https://restful.io/webhooks-dos-and-dont-s-what-we-learned-after-integrating-100-apis-d567405a3671) - * [Why Are Webhooks Better Than Serverless Extensibility?](https://developer.okta.com/blog/2017/10/11/why-are-webhooks-better-than-serverless-extensibility) * [Webhooks Provide an Efficient Alternative to API Polling](https://thenewstack.io/wonderful-world-webhooks/) diff --git a/content/pages/04-web-development/48-api-creation.markdown b/content/pages/04-web-development/48-api-creation.markdown index da38541c2..aa49c5aa1 100644 --- a/content/pages/04-web-development/48-api-creation.markdown +++ b/content/pages/04-web-development/48-api-creation.markdown @@ -26,9 +26,6 @@ applications through machine-to-machine communication. [own GitHub organization](https://github.com/flask-restful/flask-restful) so engineers from outside the company could be core contributors. -* [Flask API](http://www.flaskapi.org/) is another common library for - exposing APIs from Flask web applications. - * [Sandman](http://www.github.com/jeffknupp/sandman) is a widely used tool to automatically generate a RESTful API service from a legacy database without writing a line of code (though it's easily extensible through code). @@ -133,12 +130,8 @@ equivalent of browser testing in the web application world. compatibility and a whole slew of other great advice for developers and API designers. -* [Self-descriptive, isn't. Don't assume anything.](http://www.bizcoder.com/self-descriptive-isn-t-don-t-assume-anything) - is an appeal that metadata makes a difference in whether APIs are descriptive - or not. - * [Designing the Artsy API](http://artsy.github.io/blog/2014/09/12/designing-the-public-artsy-api/) - has their recommendations list for building an API based on their recent + has their recommendations list for building an API based on their experiences. * Hacker News had a discussion on @@ -154,10 +147,6 @@ equivalent of browser testing in the web application world. will use your API, as well as what the documentation for endpoints and other important pieces should look like. -* [1-to-1 Relationships and Subresources in REST APIs](http://developers.lyst.com/2015/02/20/1-to-1-relationships-and-subresources-in-rest-apis/) - tells the story of design decisions that were made during an API's creation - and why those choices were made. - * [How many status codes does your API need?](https://blogs.dropbox.com/developers/2015/04/how-many-http-status-codes-should-your-api-use/) gives an answer from a Dropbox API developer as to their decision making process. @@ -219,10 +208,6 @@ equivalent of browser testing in the web application world. [Serpy](https://github.com/clarkduvall/serpy) and [wrote a blog post with the results of its performance](https://engineering.betterworks.com/2015/09/04/ditching-django-rest-framework-serializers-for-serpy/). -* [Designing a Web API](http://restlet.com/company/blog/2015/03/16/api-design-designing-a-web-api/) - gives a detailed walkthrough of concepts and design decisions you need - to make when building an API. - * Microsoft's [REST API Guidelines](https://github.com/Microsoft/api-guidelines/blob/master/Guidelines.md) are a detailed set of considerations for when you are building your own diff --git a/content/pages/04-web-development/50-django-rest-framework-drf.markdown b/content/pages/04-web-development/50-django-rest-framework-drf.markdown index 5824966bc..8e37986e0 100644 --- a/content/pages/04-web-development/50-django-rest-framework-drf.markdown +++ b/content/pages/04-web-development/50-django-rest-framework-drf.markdown @@ -60,10 +60,6 @@ typically abbreviated "DRF", is a Python library for building web * [Optimizing slow Django REST Framework performance](https://ses4j.github.io/2015/11/23/optimizing-slow-django-rest-framework-performance/) -* [TLT: Serializing Authenticated User Data With Django REST Framework](http://gregblogs.com/tlt-serializing-authenticated-user-data-with-django-rest-framework/) - -* [Building an API with Django REST Framework and Class-Based Views](https://codeburst.io/building-an-api-with-django-rest-framework-and-class-based-views-75b369b30396) - * [Simple Nested API Using Django REST Framework](https://blog.apptension.com/2017/09/13/rest-api-using-django-rest-framework/) * [Building APIs with Django and Django Rest Framework](https://books.agiliq.com/projects/django-api-polls-tutorial/en/latest/) diff --git a/content/pages/04-web-development/51-api-integration.markdown b/content/pages/04-web-development/51-api-integration.markdown index 09c6e8099..ee646bfa9 100644 --- a/content/pages/04-web-development/51-api-integration.markdown +++ b/content/pages/04-web-development/51-api-integration.markdown @@ -63,9 +63,6 @@ across many implementation areas. is a nice tutorial for easily re-executing failed HTTP requests with the Requests library. -* My DjangoCon 2013 talk dove into - "[Making Django Play Nice With Third Party Services](http://www.youtube.com/watch?v=iGP8DQIqxXs)." - * If you're looking for a fun project that uses two web APIs within a Django application, try out this tutorial to [Build your own Pokédex with Django, MMS and PokéAPI](https://www.twilio.com/blog/2014/11/build-your-own-pokedex-with-django-mms-and-pokeapi.html). diff --git a/content/pages/04-web-development/52-twilio.markdown b/content/pages/04-web-development/52-twilio.markdown index 9ba51f456..e038e3907 100644 --- a/content/pages/04-web-development/52-twilio.markdown +++ b/content/pages/04-web-development/52-twilio.markdown @@ -77,10 +77,6 @@ for fellow developers. [Twilio SMS API via some Python code](https://www.twilio.com/docs/quickstart/python/sms) to send a text message with the results. -* IBM's Bluemix blog contains a nice tutorial on building an - [IoT Python app with a Raspberry Pi and Bluemix](https://developer.ibm.com/bluemix/2015/04/02/tutorial-using-a-raspberry-pi-python-iot-twilio-bluemix/) - that uses Twilio to interact with the Raspberry Pi. - * The [Python tag on the Twilio blog](https://www.twilio.com/blog/tag/python) provides walkthroughs for [Django](https://www.twilio.com/blog/2015/12/city-chat-with-python-django-and-twilio-ip-messaging.html), diff --git a/content/pages/04-web-development/53-stripe.markdown b/content/pages/04-web-development/53-stripe.markdown index 9e5a2dd9b..82aba71f0 100644 --- a/content/pages/04-web-development/53-stripe.markdown +++ b/content/pages/04-web-development/53-stripe.markdown @@ -23,9 +23,6 @@ for processing payments. the subscription data in the [Django ORM](/django-orm.html) and create a pricing page. -* [Switching from Braintree to Stripe](https://www.deekit.com/braintree-to-stripe/) - covers one development team's experience with moving payment providers. - * [Dirt Cheap Recurring Payments with Stripe and AWS Lambda](http://normal-extensions.com/2017/05/05/simple-recurring/) explains how to use the Stripe API with [AWS Lambda](/aws-lambda.html) to handle recurring payments instead of using a more expensive service @@ -43,6 +40,11 @@ for processing payments. ### Resources about Stripe +* [Stripe’s payments APIs: the first ten years](https://stripe.com/blog/payment-api-design) + has a ton of great context about how Stripe's payments API has evolved, + its architecture, how they expanded it over time, and generally a bunch + of solid storytelling behind how it has been built. + * [How Stripe Designs Beautiful Websites](https://leerob.io/blog/how-stripe-designs-beautiful-websites) explains the process for how Stripe creates their gorgeous design that makes people want to use the service and explore what else they can diff --git a/content/pages/04-web-development/56-web-app-security.markdown b/content/pages/04-web-development/56-web-app-security.markdown index 2b1ee70f9..992091dbf 100644 --- a/content/pages/04-web-development/56-web-app-security.markdown +++ b/content/pages/04-web-development/56-web-app-security.markdown @@ -14,8 +14,6 @@ request forgery and usage of public-private keypairs. ### Security tools -* [Bro](http://www.bro.org/) is a network security and traffic monitor. - * [lynis](https://cisofy.com/lynis/) ([source code](https://github.com/CISOfy/lynis)) is a security audit tool that can run as a shell script on a Linux system to find out @@ -112,7 +110,7 @@ resources can also give you a good overview of how HTTPS works. learn about web browser internals, session attacks, fingerprinting, HTTPS and many other fundamental topics. -* [The SaaS CTO Security Checklist](https://cto-security-checklist.sqreen.io/) +* [The SaaS CTO Security Checklist Redux](https://www.goldfiglabs.com/guide/saas-cto-security-checklist/) is an awesome list of steps for securing your infrastructure and employees as well as what stage and size company it is recommended that you put those procedures in place. @@ -132,11 +130,6 @@ resources can also give you a good overview of how HTTPS works. * The [/r/netsec](http://www.reddit.com/r/netsec/) subreddit is one place to go to learn more about network and application security. -* [Hacking Tools Repository](http://gexos.github.io/Hacking-Tools-Repository/) - is a great list of password cracking, scanning, sniffing and other security - penetration testing tools. - - * The EFF has a well written overview on [what makes a good security audit](https://www.eff.org/deeplinks/2014/11/what-makes-good-security-audit). It's broad but contains some of their behind the scenes thinking on important considerations with security audits. @@ -160,10 +153,6 @@ resources can also give you a good overview of how HTTPS works. important technique to use to keep your database passwords and other secrets more secure if the hashed strings are leaked. -* [An in-depth analysis of SSH attacks on Amazon EC2](http://getprismatic.com/story/1409447605839) - shows how important it is to secure your web servers, especially when they are - hosted in IP address ranges that are commonly scanned by malicious actors. - * [Cloud Security Auditing: Challenges and Emerging Approaches](http://www.infoq.com/articles/cloud-security-auditing-challenges-and-emerging-approaches) is a high-level overview of some of security auditing problems that come with cloud deployments. diff --git a/content/pages/04-web-development/57-sql-injection.markdown b/content/pages/04-web-development/57-sql-injection.markdown index 8536fdb73..084f699e8 100644 --- a/content/pages/04-web-development/57-sql-injection.markdown +++ b/content/pages/04-web-development/57-sql-injection.markdown @@ -24,3 +24,8 @@ can affect both [relational databases](/databases.html) and * [Securing your site like it's 1999](https://24ways.org/2018/securing-your-site-like-its-1999/) covers a bunch of common web application vulnerabilities including SQL injection. + +* [Automating Blind Sql Injection](https://bad-jubies.github.io/Blind-SQLi-1/) + shows how to use Python to execute SQL injection on the example + [Damn Vulnerable Web Application](https://github.com/digininja/DVWA) + project. diff --git a/content/pages/05-deployment/00-deployment.markdown b/content/pages/05-deployment/00-deployment.markdown index a7080da8a..48ad4e8c1 100644 --- a/content/pages/05-deployment/00-deployment.markdown +++ b/content/pages/05-deployment/00-deployment.markdown @@ -51,7 +51,7 @@ guide as they are considered advanced deployment techniques. * [teletraan](https://github.com/pinterest/teletraan) is the deploy system used by the development teams at Pinterest, a huge Python shop! -* [pants](https://www.pantsbuild.org/index.html) is a build system originally +* [pants](https://www.pantsbuild.org/) is a build system originally created at Twitter and now split out as its own sustainable open source project. @@ -132,14 +132,6 @@ guide as they are considered advanced deployment techniques. is an awesome in-depth read covering topics ranging from git branching to database migrations. -* In [this free video by Neal Ford](http://player.oreilly.com/videos/9781491908181?toc_id=210188), - he talks about engineering practices for continuous delivery. He explains - the difference between - [continuous integration](/continuous-integration.html), - continuous deployment and continuous delivery. Highly recommended for an - overview of deployment concepts and as an introduction to the other videos - on those subjects in that series. - * [TestDriven.io](https://testdriven.io/) shows how to deploy a [microservices](/microservices.html) architecture that uses [Docker](/docker.html), [Flask](/flask.html), and React with diff --git a/content/pages/05-deployment/02-servers.markdown b/content/pages/05-deployment/02-servers.markdown index 90bb07951..f66f68977 100644 --- a/content/pages/05-deployment/02-servers.markdown +++ b/content/pages/05-deployment/02-servers.markdown @@ -128,8 +128,8 @@ provides a unified API for many cloud service providers. * [Amazon Web Services has official documentation](http://aws.amazon.com/python/) for running Python web applications. -* [boto](https://github.com/boto/boto) is an extensive and well-tested -Python library for working with Amazon Web Services. +* [boto3](https://github.com/boto/boto3) is an extensive and well-tested + Python library for working with Amazon Web Services. * [Poseidon](https://github.com/changhiskhan/poseidon) is a Python commandline interface for managing Digital Ocean droplets (servers). diff --git a/content/pages/05-deployment/03-static-content.markdown b/content/pages/05-deployment/03-static-content.markdown index 8c6e88262..f5a118f54 100644 --- a/content/pages/05-deployment/03-static-content.markdown +++ b/content/pages/05-deployment/03-static-content.markdown @@ -21,9 +21,9 @@ Django framework calls these two categories *assets* and *media*. ## Content delivery networks A content delivery network (CDN) is a third party that stores and serves static files. [Amazon CloudFront](http://aws.amazon.com/cloudfront/), -[Akamai](http://www.akamai.com/), and -[Rackspace Cloud Files](http://www.rackspace.com/cloud/public/files/) -are examples of CDNs. The purpose of a CDN is to remove the load of static +[CloudFlare](https://www.cloudflare.com/) and [Fastly](https://www.fastly.com/), +are examples of CDN services. The purpose of +a CDN is to remove the load of static file requests from web servers that are handling dynamic web content. For example, if you have an nginx server that handles both static files and acts as a front for a Green Unicorn WSGI server on a 512 megabyte diff --git a/content/pages/05-deployment/04-cdns.markdown b/content/pages/05-deployment/04-cdns.markdown index 027f064d6..bd4e516fe 100644 --- a/content/pages/05-deployment/04-cdns.markdown +++ b/content/pages/05-deployment/04-cdns.markdown @@ -12,9 +12,9 @@ servers to improve web app loading speed. ### CDN resources -* [Mastering HTTP Caching](https://blog.fortrabbit.com/mastering-http-caching) - is a fantastic post that goes into great technical detail on how CDNs and - caching work. +* [The 5 hour CDN](https://fly.io/blog/the-5-hour-content-delivery-network/) + explains the basics of what CDNs are and how they are a combination of + many standard web server components, but used globally and at scale. * [MaxCDN vs CloudFlare vs Amazon CloudFront vs Akamai Edge vs Fastly](https://www.codeinwp.com/blog/maxcdn-vs-cloudflare-vs-cloudfront-vs-akamai-edge-vs-fastly/) compares and contrasts the most popular CDN services based on features, @@ -29,10 +29,6 @@ servers to improve web app loading speed. [django-storages](https://django-storages.readthedocs.io/en/latest/) library to deploy static assets for a [Django](/django.html) application to a CDN. -* [Building your own CDN for Fun and Profit](https://pasztor.at/blog/building-your-own-cdn) - is a great high-level overview of how CDNs work and shows you how to - create your own, albeit simplified CDN. - * [Do not let your CDN betray you: Use Subresource Integrity](https://hacks.mozilla.org/2015/09/subresource-integrity-in-firefox-43/) describes the security implications for CDNs with unexpectedly modified content and how Subresource Integrity in modern web browsers can mitigate diff --git a/content/pages/05-deployment/09-paas.markdown b/content/pages/05-deployment/09-paas.markdown index 9416e971f..c89a37439 100644 --- a/content/pages/05-deployment/09-paas.markdown +++ b/content/pages/05-deployment/09-paas.markdown @@ -69,11 +69,6 @@ of controlling and modifying the project for your own applications, but prevents you from offloading the responsibility of keeping servers running to someone else. -* [Kel](http://www.kelproject.com/) uses Kubernetes as a foundation - for a custom self-hosted PaaS. Note that it was created by Eldarion, - which had one of the first Python-specific PaaS offerings on the - market around the time that Heroku was launched. - * [Dokku](http://dokku.viewdocs.io/dokku/) builds on Docker and has hooks for plugins to extend the small core of the project and customize deployments for your applications. @@ -82,7 +77,6 @@ running to someone else. designed to run on top of AWS services. - ## Platform-as-a-service resources * [The differences between IaaS, PaaS and SaaS](https://www.engineyard.com/blog/the-differences-between-iaas-paas-and-saas-and-when-to-use-each) explains the abstract layer differences among "X-as-a-service" offering @@ -156,11 +150,6 @@ running to someone else. on Google Cloud and posits what they may be paying to run their service. -* [PaaS (false) economics](https://blog.drie.co/paas-false-economics-13f72d87b485) - gives some quick back-of-the-envelope calculations on why running your - applications on a PaaS is obviously going to appear more expensive if you - do not take the cost of your own software engineers into the equation. - * Two blog posts on using AWS Autoscaling in [Automatic replacement of Autoscaling nodes with equivalent spot instances](https://mcristi.wordpress.com/2016/04/21/my-approach-at-making-aws-ec2-affordable-automatic-replacement-of-autoscaling-nodes-with-equivalent-spot-instances/) and [Autoscaling nodes: seeing it in action](https://mcristi.wordpress.com/2016/04/27/automatic-replacement-of-autoscaling-nodes-with-equivalent-spot-instances-seeing-it-in-action/) diff --git a/content/pages/05-deployment/10-heroku.markdown b/content/pages/05-deployment/10-heroku.markdown index 63aad7441..06c1d7bcb 100644 --- a/content/pages/05-deployment/10-heroku.markdown +++ b/content/pages/05-deployment/10-heroku.markdown @@ -21,7 +21,14 @@ easily [deploy](/deployment.html) Python applications. [MySQL](/mysql.html) to [PostgreSQL](/postgresql.html) if necessary as well as how to properly handle your settings files. -* Heroku's +* [How to deploy Django project to Heroku using Docker](https://www.accordbox.com/blog/deploy-django-project-heroku-using-docker/) + explains that although [Buildpacks](https://devcenter.heroku.com/articles/buildpacks) + are the most common way to deploy to Heroku, packaing your app in a + [Docker](/docker.html) container is also a viable approach. It walks through + the steps needed to deploy a [Django](/django.html) app in the remainder + of the article. + +* Heroku's [official Python documentation](https://devcenter.heroku.com/articles/getting-started-with-python) is fantastic and walks through deploying WSGI applications in short order. diff --git a/content/pages/05-deployment/19-nginx.markdown b/content/pages/05-deployment/19-nginx.markdown index afe620d84..3f17e6524 100644 --- a/content/pages/05-deployment/19-nginx.markdown +++ b/content/pages/05-deployment/19-nginx.markdown @@ -63,7 +63,7 @@ to make sure you are avoiding the most common security errors that plague HTTP(S) configurations. * [HTTPS with Let's Encrypt and nginx](https://botleg.com/stories/https-with-lets-encrypt-and-nginx/) - walks throough installing a free SSL certificate from Let's Encrypt + walks through installing a free SSL certificate from Let's Encrypt to secure HTTP connects to your nginx server via HTTPS. * The [Nginx Config](https://nginxconfig.io/) tool can generate strong @@ -74,9 +74,7 @@ HTTP(S) configurations. * [Strong SSL Security on Nginx](https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html) shows how to mitigate high profile SSL attacks like - [Logjam](https://weakdh.org/), - [Heartbleed](http://heartbleed.com/) - and [FREAK](https://freakattack.com/). + [Logjam](https://weakdh.org/) and [Heartbleed](http://heartbleed.com/). ## Specific Nginx resources @@ -103,7 +101,7 @@ several years. * [Test-driving web server configuration](https://gdstechnology.blog.gov.uk/2015/03/25/test-driving-web-server-configuration/) tells a good story for how to iteratively apply configuration changes, such - as routing traffic to [Piwik](http://piwik.org/) for + as routing traffic to [Matoma](https://matomo.org/) for [web analytics](/web-analytics.html), reverse proxying to backend application servers and terminately TLS connections appropriately. It is impressive to read a well-written softare development article like @@ -118,10 +116,6 @@ several years. as well as the Pagespeed module that Google released for both Nginx and the [Apache HTTP Server](/apache-http-server.html). -* [Nginx for Developers: An Introduction](http://carrot.is/coding/nginx_introduction) - provides the first steps to getting an initial Nginx configuration up and - running. - * [A faster Web server: ripping out Apache for Nginx](http://arstechnica.com/business/2011/11/a-faster-web-server-ripping-out-apache-for-nginx/) explains how Nginx can be used instead of Apache in some cases for better performance. diff --git a/content/pages/05-deployment/20-caddy.markdown b/content/pages/05-deployment/20-caddy.markdown index 1b27e6d11..56ca43167 100644 --- a/content/pages/05-deployment/20-caddy.markdown +++ b/content/pages/05-deployment/20-caddy.markdown @@ -15,7 +15,8 @@ and design emphasize HTTPS-everywhere along with the HTTP/2 protocol. ## How can Caddy be used with Python deployments? Caddy can be used both for testing during local development or as part of a production deployment as an HTTP server and a reverse proxy with -the [proxy directive](https://caddyserver.com/docs/proxy). +the +[reverse_proxy directive](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy).
Caddy is an implementation of the web server concept. Learn how these pieces fit together in the deployment chapter or view the table of contents for all topics.
diff --git a/content/pages/05-deployment/22-wsgi-servers.markdown b/content/pages/05-deployment/22-wsgi-servers.markdown index 35174df41..036bdbe54 100644 --- a/content/pages/05-deployment/22-wsgi-servers.markdown +++ b/content/pages/05-deployment/22-wsgi-servers.markdown @@ -177,9 +177,9 @@ The following are WSGI servers based on community recommendations. is a good read to understand basic information about various WSGI server implementations. -* A thorough and informative post for LAMP-stack hosting choices is - presented in the - "[complete single server Django stack tutorial](http://www.apreche.net/complete-single-server-django-stack-tutorial/)." +* [What is WSGI and Why Do You Need Gunicorn and Nginx in Django](https://apirobot.me/posts/what-is-wsgi-and-why-do-you-need-gunicorn-and-nginx-in-django) + explains the breakdown between a [web server](/web-servers.html) + and a WSGI server in an application deployment environment. * The Python community made a long effort to [transition from mod\_python](http://blog.dscpl.com.au/2010/05/modpython-project-soon-to-be-officially.html) diff --git a/content/pages/05-deployment/23-gunicorn.markdown b/content/pages/05-deployment/23-gunicorn.markdown index f7098a1b4..1973ab6ff 100644 --- a/content/pages/05-deployment/23-gunicorn.markdown +++ b/content/pages/05-deployment/23-gunicorn.markdown @@ -49,7 +49,7 @@ file with the following contents: It exposes the WSGI callable as a module-level variable named ``application``. For more information on this file, see - https://docs.djangoproject.com/en/1.8/howto/deployment/wsgi/ + https://docs.djangoproject.com/en/3.2/howto/deployment/wsgi/ """ import os @@ -115,11 +115,15 @@ perform the request handling. Each worker is independent of the controller. screenshots along the way with what to expect while you are configuring the deployment server. -* The [Django](https://docs.djangoproject.com/en/1.9/howto/deployment/wsgi/gunicorn/) +* The [Django](https://docs.djangoproject.com/en/3.2/howto/deployment/wsgi/gunicorn/) and [Flask](http://flask.pocoo.org/docs/latest/deploying/wsgi-standalone/) documentation each contain instructions for deploying the respective frameworks with Gunicorn. +* [Dockerizing Django with Postgres, Gunicorn, and Traefik](https://testdriven.io/blog/django-docker-traefik/) + is a more advanced tutorial with a complete project deployment + that uses Gunicorn within [Docker](/docker.html). + * [Set up Django, Nginx and Gunicorn in a Virtualenv controled by Supervisor](https://gist.github.com/Atem18/4696071) is a GitHub Gist with some great explanations for why we're setting up virtualenv and what to watch out for while you're doing the deployment. diff --git a/content/pages/05-deployment/24-uwsgi.markdown b/content/pages/05-deployment/24-uwsgi.markdown index 985d326fe..7561de3da 100644 --- a/content/pages/05-deployment/24-uwsgi.markdown +++ b/content/pages/05-deployment/24-uwsgi.markdown @@ -42,7 +42,3 @@ implementation that is typically used to run Python web applications. is awesome because it shows you how to code a quick WSGI application without using a framework then builds up an example with deploying a traditional Django web app. - -* [Deployment Notes for Pylons, Nginx, and uWSGI](http://tonylandis.com/python/deployment-howt-pylons-nginx-and-uwsgi/) - gives the code and instructions for setting up a Pylons application - with uWSGI. diff --git a/content/pages/05-deployment/27-continuous-integration.markdown b/content/pages/05-deployment/27-continuous-integration.markdown index 835f5589a..ea620e337 100644 --- a/content/pages/05-deployment/27-continuous-integration.markdown +++ b/content/pages/05-deployment/27-continuous-integration.markdown @@ -127,11 +127,6 @@ programming language agnostic. Learn more via the following resources or on uses well done drawings to show how continuous integration and delivery works for testing and managing data. -* [The real difference between CI and CD](https://fire.ci/blog/the-difference-between-ci-and-cd/) - explains what advantages CI provides, what constraints it operates under - (such as total build time) to work well, and how that is different from - the related but distinct concept of continuous delivery. - * [6 top continuous integration tools](https://opensource.com/business/15/7/six-continuous-integration-tools) gives a high level overview of six CI tools from a programming language agnostic perspective. diff --git a/content/pages/05-deployment/28-jenkins.markdown b/content/pages/05-deployment/28-jenkins.markdown index 46a711870..63c64d57c 100644 --- a/content/pages/05-deployment/28-jenkins.markdown +++ b/content/pages/05-deployment/28-jenkins.markdown @@ -35,20 +35,11 @@ used to automate building, [testing](/testing.html) and is another solid tutorial that also shows how to send email notifications as part of the build process. -* If you're running into difficulty adding an SSH key to your Jenkins system - account so you can connect to another server or Git repository - [this blog post on connecting Jenkins with Git](http://dcycleproject.org/blog/51/connecting-jenkins-and-git) - to get the steps to solve that problem. - * [Running Jenkins in Docker Containers](http://www.catosplace.net/blog/2015/02/11/running-jenkins-in-docker-containers/) is a short tutorial showing how to use the official [Jenkins container](https://registry.hub.docker.com/_/jenkins/) on the Docker hub. -* [Securing Jenkins](https://wiki.jenkins.io/display/JENKINS/Securing+Jenkins) - is the landing page for Jenkins security. If you're deploying your own - instance, you'll need to lock it down against unauthorized users. - * [Updating the GOV.UK Continuous Integration environment](https://gdstechnology.blog.gov.uk/2017/02/10/updating-the-gov-uk-continuous-integration-environment/) describes the configuration improvements one infrastructure team made to Jenkins, where they enabled @@ -77,7 +68,3 @@ used to automate building, [testing](/testing.html) and * [Automated API testing with Jenkins](https://assertible.com/blog/automated-api-testing-with-jenkins) walks through how to use Jenkins to tests your [API](/application-programming-interfaces.html) upon each deployment. - -* [Continuous Delivery with Jenkins and Rollbar](https://rollbar.com/blog/continuous-delivery-with-jenkins/) - is a tutorial on using Jenkins for continuous integration paired with - [Rollbar](/rollbar.html) for tracking deployments and errors. diff --git a/content/pages/05-deployment/32-configuration-management.markdown b/content/pages/05-deployment/32-configuration-management.markdown index 252df484e..23f4b07ba 100644 --- a/content/pages/05-deployment/32-configuration-management.markdown +++ b/content/pages/05-deployment/32-configuration-management.markdown @@ -32,10 +32,6 @@ operations, such as querying the database from the Django manage.py shell. is an openly biased but detailed post on why to choose SaltStack over Ansible in certain situations. -* [Ansible vs. Shell Scripts](https://valdhaus.co/writings/ansible-vs-shell-scripts/) - provides some perspective on why using a configuration management tool is a - better choice than venerable but brittle shell scripts. - * [Ansible vs. Chef](http://tjheeta.github.io/2015/04/15/ansible-vs-chef/) is a comparsion of Ansible with the Chef configuration management tool. @@ -59,17 +55,6 @@ management and application deployment tool built in Python. is a fantastically detailed introduction on using Ansible to set up servers. -* [Ansible Text Message Notifications with Twilio SMS](https://www.twilio.com/blog/2014/05/ansible-text-messages-notifications-with-twilio-sms.html) - is my blog post with a detailed example for using the Twilio module in - core Ansible 1.6+. - -* [Python for Configuration Management with Ansible slides](http://www.insom.me.uk/post/pycon-talk.html) -from PyCon UK 2013 - -* [First Steps with Ansible](http://labs.qandidate.com/blog/2013/11/15/first-steps-with-ansible/) - -* [Red Badger on Ansible](http://red-badger.com/blog/2013/06/29/ansible/) - * [Getting Started with Ansible](http://lowendbox.com/blog/getting-started-with-ansible/) * [An introduction to Ansible](https://davidwinter.me/introduction-to-ansible/) @@ -89,8 +74,6 @@ from PyCon UK 2013 * [Idempotence, convergence, and other silly fancy words we often use](https://groups.google.com/forum/#!msg/Ansible-project/WpRblldA2PQ/lYDpFjBXDlsJ) -* [Testing with Jenkins, Docker and Ansible](http://blog.mist.io/post/82383668190/move-fast-and-dont-break-things-testing-with) - ## Application dependencies learning checklist 1. Learn about configuration management in the context of deployment diff --git a/content/pages/05-deployment/33-ansible.markdown b/content/pages/05-deployment/33-ansible.markdown index a64409f26..bbdff142a 100644 --- a/content/pages/05-deployment/33-ansible.markdown +++ b/content/pages/05-deployment/33-ansible.markdown @@ -38,9 +38,6 @@ be able to structure your playbooks: configures macOS with various applications and developer tools such as [Docker](/docker.html), Homebrew and [Sublime Text](/sublime-text.html). -* [ansible-nginx-haproxy-elasticsearch](https://github.com/gp187/ansible-nginx-haproxy-elasticsearch) - sets up a server with [Nginx](/nginx.html), HAProxy and ElasticSearch. - ### Specific Ansible topics * [An Ansible2 Tutorial](https://serversforhackers.com/c/an-ansible2-tutorial) @@ -63,10 +60,6 @@ be able to structure your playbooks: * [DevOps from Scratch, Part 1: Vagrant & Ansible](https://www.kevinlondon.com/2016/09/19/devops-from-scratch-pt-1.html) -* [Ansible: Post-Install Setup](https://valdhaus.co/writings/ansible-post-install/) - -* [How To Use Vault to Protect Sensitive Ansible Data on Ubuntu 16.04](https://www.digitalocean.com/community/tutorials/how-to-use-vault-to-protect-sensitive-ansible-data-on-ubuntu-16-04) - * [How to use Ansible Variables and Vaults](https://www.expressvpn.com/blog/ansible-variables-vaults/) * [CI for Ansible playbooks which require Ansible Vault protected variables](https://www.jeffgeerling.com/blog/2017/ci-ansible-playbooks-which-require-ansible-vault-protected-variables) diff --git a/content/pages/05-deployment/35-containers.markdown b/content/pages/05-deployment/35-containers.markdown index f88e53a02..5ec0a850e 100644 --- a/content/pages/05-deployment/35-containers.markdown +++ b/content/pages/05-deployment/35-containers.markdown @@ -70,6 +70,12 @@ useful. explains how Linux features such as `cgroups`, `chroot` and namespaces are used by container implementations. +* [Container networking is simple](https://iximiuz.com/en/posts/container-networking-is-simple/) + shows that container networking is nothing more than a simple combination + of the well-known Linux facilities such as network namespaces, virtual + Ethernet devices (veth), virtual network switches (bridge) and + IP routing and network address translation (NAT). + * [Running containers without Docker](https://jvns.ca/blog/2016/10/26/running-container-without-docker/) reviews a migration path for an organization that already has a bunch of infrastructure but sees advantages in using containers. However, the @@ -77,6 +83,11 @@ useful. eventually plan to use Docker, Kubernetes or other container tools and orchestration layer. +* [Datadog's 2020 Container Report](https://www.datadoghq.com/container-report/) + contains some interesting statistics about container usage across + their customer base, such as [Kubernetes](/kubernetes.html) adoption + and container deployments by cloud platform. + * [mocker](https://github.com/tonybaloney/mocker) is a Docker imitation open source project written in all Python which is intended for learning purposes. @@ -115,6 +126,13 @@ Container security is a hot topic because there are so many ways of screwing it up, just like any infrastructure that runs your applications. These resources explain security considerations specific to containers. +* [A Practical Introduction to Container Security](https://cloudberry.engineering/article/practical-introduction-container-security/) + examines security at build time for projects and how to + minimize the risk of supply chain attack. It then goes into + infrastructure and runtime security where you need to understand + different attack vectors and minimize malicious attempts against + your containers during these phases.. + * [Building Container Images Securely on Kubernetes](https://blog.jessfraz.com/post/building-container-images-securely-on-kubernetes/) discusses some of the issues with building containers and why the author created [img](https://github.com/genuinetools/img) as a tool diff --git a/content/pages/05-deployment/36-docker.markdown b/content/pages/05-deployment/36-docker.markdown index 8017a95da..f133b70f0 100644 --- a/content/pages/05-deployment/36-docker.markdown +++ b/content/pages/05-deployment/36-docker.markdown @@ -48,20 +48,10 @@ on Amazon Web Services, Google Compute Engine, Linode, Rackspace or elsewhere. repository and tutorial that shows you how to recreate a simplified version of Docker to better understand what it's doing under the hood. -* [Andrew Baker](https://github.com/atbaker) presented a fantastic tutorial - at [PyOhio](http://andrewtorkbaker.com/pyohio-docker-101-tutorial) on - [beginner and advanced Docker usage](https://github.com/atbaker/docker-tutorial). - Andrew also wrote the article - [what containers can do for you](http://radar.oreilly.com/2015/01/what-containers-can-do-for-you.html). - * [Docker curriculum](http://prakhar.me/docker-curriculum/) is a detailed tutorial created by a developer to show the exact steps for deploying an application that relies on [Elasticsearch](https://www.elastic.co/). -* [How To Install and Use Docker on Ubuntu 16.04](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-16-04) - provides a walkthrough for Ubuntu 16.04 for installing and beginning to - use Docker for development. - * [It Really is the Future](http://blog.circleci.com/it-really-is-the-future/) discusses Docker and containers in the context of whether it's all just a bunch of hype or if this is a real trend for infrastructure automation. @@ -87,10 +77,6 @@ on Amazon Web Services, Google Compute Engine, Linode, Rackspace or elsewhere. is a short tutorial for creating a Docker container with a specific configuration. -* [10 things to avoid in Docker containers](http://developerblog.redhat.com/2016/02/24/10-things-to-avoid-in-docker-containers/) - provides a lot of "don'ts" that you'll want to consider before bumping - up against the limitations of how containers should be used. - * [Docker Internals](http://docker-saigon.github.io/post/Docker-Internals/) presents Linux containers and how Docker uses them as its base for how the project works. This article is a great way to bridge what you know about Docker with a more diff --git a/content/pages/05-deployment/37-kubernetes.markdown b/content/pages/05-deployment/37-kubernetes.markdown index 3e029b97d..96e69ef62 100644 --- a/content/pages/05-deployment/37-kubernetes.markdown +++ b/content/pages/05-deployment/37-kubernetes.markdown @@ -30,11 +30,11 @@ applications. * [Kompose](http://kompose.io/index) ([source code](https://github.com/kubernetes/kompose)) - translate Docker Compose files into Kubernetes configuration resources. + translates Docker Compose files into Kubernetes configuration resources. -* [skaffold](https://skaffold.dev/). [Using Kubernetes for local development](https://nemethgergely.com/using-kubernetes-for-local-development/index.html) - is a good starting place for more information on getting started with - Skaffold. +* [skaffold](https://skaffold.dev/) + ([source code](https://github.com/GoogleContainerTools/skaffold)) makes + it easier to develop locally with Kubernetes. * [kubethanos](https://github.com/berkay-dincer/kubethanos) is a tool to kill half of your Kubernetes pods at random, to test the resilience of your @@ -88,7 +88,7 @@ applications. cost-benefit analysis to make sure the tool's scability, reliability and related functionality will outweigh the downsides. -* [How Zolando manages 140+ Kubernetes clusters](https://srcco.de/posts/how-zalando-manages-140-kubernetes-clusters.html) +* [How Zalando manages 140+ Kubernetes clusters](https://srcco.de/posts/how-zalando-manages-140-kubernetes-clusters.html) covers the architecture, monitoring and workflow of a team that has to run a decent number of clusters for their development teams. diff --git a/content/pages/05-deployment/38-serverless.markdown b/content/pages/05-deployment/38-serverless.markdown index 306caa320..a5236c280 100644 --- a/content/pages/05-deployment/38-serverless.markdown +++ b/content/pages/05-deployment/38-serverless.markdown @@ -45,9 +45,8 @@ These implementations are under significant active development and not all of them have Python support. * [AWS Lambda](/aws-lambda.html) is the current leader among serverless - compute implementations. It has support for both - [Python 2.7](/blog/aws-lambda-python-2-7.html) and - [Python 3.6/3.7](/blog/aws-lambda-python-3-6.html). + compute implementations. It has support for + [Python 3.x](/blog/aws-lambda-python-3-6.html). * Azure Functions has second-class citizen support for Python. It's supposed to be possible but @@ -59,14 +58,11 @@ and not all of them have Python support. [Apache OpenWhisk](https://github.com/openwhisk/openwhisk) open source project. -* [Google Cloud Functions](/google-cloud-functions.html) currently - only supports JavaScript code execution. +* [Google Cloud Functions](/google-cloud-functions.html) has + [native Python 3.x runtimes](https://cloud.google.com/functions/docs/concepts/python-runtime). -* Webtask.io also only supports JavaScript but there is a cool - *prototype* project named [webtask-pytask](https://github.com/tehsis/webtask-pytask) - to run Python code in the browser via webtask. This demo is definitely not - for production code use but awesome to see what the programming community - can put together using existing code and services. +* [Webtask.io](https://webtask.io/) started as a JavaScript service but + now also has a Python runtime as well. ### Serverless frameworks @@ -83,8 +79,7 @@ include: which is a useful but generically-named library that focuses on deployment and operations for serverless applications. -* [Zappa](https://www.zappa.io/) - ([source code](https://github.com/Miserlou/Zappa)) +* [Zappa](https://github.com/Miserlou/Zappa) provides code and tools to make it much easier to build on AWS Lambda and AWS API Gateway than rolling your own on the bare services. @@ -92,8 +87,6 @@ include: ([source code](https://github.com/aws/chalice)) is built by the AWS team specifically for Python applications. -* [Apex](http://apex.run/) ([source code](https://github.com/apex/apex)) - ### General serverless resources Serverless concepts and implementations are still in their early @@ -101,6 +94,12 @@ iterations so there are many ideas and good practices yet to be discovered. These resources are the first attempts at figuring out how to structure and operate serverless applications. +* [What's Serverless?](https://technically.substack.com/p/whats-serverless) + is an accessible "first read" for both developers and non-technical + audiences alike. It breaks down the differences between what most + developers consider serverless and infrastructure-as-a-service (IaaS) + offerings. + * [Serverless software](https://talkpython.fm/episodes/show/118/serverless-software) covers a range of topics under serverless and how deployments have changed as new options such as [PaaS](/platform-as-a-service.html) @@ -183,37 +182,18 @@ have varying degrees of support for Python. AWS Lambda has production-ready support for Python 2 and 3.7, while Azure and Google Cloud have "beta" support with unclear production-worthiness. The following resources are some comparison articles to help you in your decision-making -process for which platform to learn. - -* [Serverless at scale](https://blog.binaris.com/serverless-at-scale/) - compares the "Big 3" AWS, Azure and Google Cloud in serverless performance. - The author provides some nice data around average response times and - outliers. - -* [Serverless hosting comparison](https://headmelted.com/serverless-showdown-4a771ca561d2) - is a broad overview of documentation, community, pricing and other - notes for the major platforms as well as IBM OpenWhisk and - the [Fission.io](https://fission.io/) project. - -* [Microsoft Azure Functions vs. Google Cloud Functions vs. AWS Lambda](https://cloudacademy.com/blog/microsoft-azure-functions-vs-google-cloud-functions-fight-for-serverless-cloud-domination-continues/) - presents an overview of Azure Functions and how they compare to - Google Cloud Functions and AWS Lambda. +process for which platform to learn. +[Microsoft Azure Functions vs. Google Cloud Functions vs. AWS Lambda](https://cloudacademy.com/blog/microsoft-azure-functions-vs-google-cloud-functions-fight-for-serverless-cloud-domination-continues/) +presents an overview of Azure Functions and how they compare to +Google Cloud Functions and AWS Lambda. ### Serverless vendor lock-in? There is some concern by organizations and developers about vendor lock-in on serverless platforms. It is unclear if portability is worse for serverless than other infrastructure-as-a-service pieces, but still worth -thinking about ahead of time. These resources provide additional -perspectives on lock-in and using multiple cloud providers. - -* [On Serverless, Multi-Cloud, and Vendor Lock In](https://blog.symphonia.io/on-serverless-multi-cloud-and-vendor-lock-in-da930b3993f) - is an opinion piece that for *most* cases the additional work of - going multi-cloud is not worth the tradeoffs, therefore at this time - it's better to go for a single vendor such as AWS or Azure and optimize - on that platform. - -* [Why vendor lock-in with serverless isn’t what you think it is](https://medium.com/@PaulDJohnston/why-vendor-lock-in-with-serverless-isnt-what-you-think-it-is-d6be40fa9ca9) - recommends using a single vendor for now and stop worrying about - hedging your bets because it typically makes your infrastructure - significantly more complex. +thinking about ahead of time. +[Why vendor lock-in with serverless isn’t what you think it is](https://medium.com/@PaulDJohnston/why-vendor-lock-in-with-serverless-isnt-what-you-think-it-is-d6be40fa9ca9) +is a piece on this topic that recommends using a single vendor for +now and for organizations to stop worrying about hedging their bets +because it typically makes infrastructure significantly more complex. diff --git a/content/pages/05-deployment/39-aws-lambda.markdown b/content/pages/05-deployment/39-aws-lambda.markdown index fec7e2944..80efa5ef7 100644 --- a/content/pages/05-deployment/39-aws-lambda.markdown +++ b/content/pages/05-deployment/39-aws-lambda.markdown @@ -38,8 +38,7 @@ has support for both Python 2.7, 3.6 and 3.7. * [Zappa](https://github.com/Miserlou/Zappa) is a serverless framework for deploying Python web applications. It's a really slick project and used even by internal AWS developers for their own application - deployments. Be sure to [read the Zappa blog](https://blog.zappa.io/) - as well for walkthroughs and new feature announcements. + deployments. * [How to Setup a Serverless URL Shortener With API Gateway Lambda and DynamoDB on AWS](https://blog.ruanbekker.com/blog/2018/11/30/how-to-setup-a-serverless-url-shortener-with-api-gateway-lambda-and-dynamodb-on-aws/) builds a non-trivial URL shortener application as an example Python @@ -53,10 +52,6 @@ has support for both Python 2.7, 3.6 and 3.7. provides a screen capture of one developer deploying their application to Lambda. -* [Automated SQL Injection Testing of Serverless Functions On a Shoestring Budget (and Some Good Music)](https://www.puresec.io/blog/automated-sql-injection-testing-of-serverless-functions-on-a-shoestring-budget-and-some-good-music) - is an awesome operational security post that uses Python to test - for SQL injection vulnerabilities in serverless functions on AWS Lambda. - * [Building Scikit-Learn For AWS Lambda](https://serverlesscode.com/post/scikitlearn-with-amazon-linux-container/) follows up on the [Using Scikit-Learn In AWS Lambda](https://serverlesscode.com/post/deploy-scikitlearn-on-lamba/) @@ -70,14 +65,6 @@ has support for both Python 2.7, 3.6 and 3.7. * [Code Evaluation With AWS Lambda and API Gateway](https://realpython.com/blog/python/code-evaluation-with-aws-lambda-and-api-gateway/) shows how to develop a code evaluation API, to execute arbitrary code, with AWS Lambda and API Gateway. -* [Crawling thousands of products using AWS Lambda](https://engineering.21buttons.com/crawling-thousands-of-products-using-aws-lambda-80332e259de1) - gives a real-world example of where using Python, Selenium and - [headless Chrome](https://developers.google.com/web/updates/2017/04/headless-chrome) - on AWS Lambda could crawl thousands of pages to collect data - with each crawler running within its own Lambda Function. - -* [Going Serverless with AWS Lambda and API Gateway](http://blog.ryankelly.us/2016/08/07/going-serverless-with-aws-lambda-and-api-gateway.html) - ### General AWS Lambda resources * [Getting started with serverless on AWS](https://emshea.com/post/serverless-getting-started) @@ -85,11 +72,6 @@ has support for both Python 2.7, 3.6 and 3.7. guide created by a developer who used all of these bits to learn AWS services herself. -* [AWS Lambda Serverless Reference Architectures](http://www.allthingsdistributed.com/2016/06/aws-lambda-serverless-reference-architectures.html) - provides blueprints with diagrams of common architecture patterns that - developers use for their mobile backend, file processing, stream - processing and web application projects. - * [Security Overview of AWS Lambda](https://d1.awsstatic.com/whitepapers/Overview-AWS-Lambda-Security.pdf) (PDF file) covers their "Shared Responsibility Model" for security and compliance. Although the paper bills itself as an in-depth look at diff --git a/content/pages/05-deployment/40-azure-functions.markdown b/content/pages/05-deployment/40-azure-functions.markdown index 4cb8243e1..7f718f64a 100644 --- a/content/pages/05-deployment/40-azure-functions.markdown +++ b/content/pages/05-deployment/40-azure-functions.markdown @@ -44,6 +44,3 @@ in other Azure services. what is confusing to newcomers that hopefully will be addressed as Microsoft continues to work on their Azure platform. -* [Azure in Plain English](https://www.expeditedssl.com/azure-in-plain-english) - covers all of the Azure services and explains them because their - default names are often too vague to understand their purpose. diff --git a/content/pages/06-devops/00-devops.markdown b/content/pages/06-devops/00-devops.markdown index 47ec79be1..f6053c2e4 100644 --- a/content/pages/06-devops/00-devops.markdown +++ b/content/pages/06-devops/00-devops.markdown @@ -35,8 +35,8 @@ tools and services for DevOps environments. which when used properly can enable continuous software delivery. * For an Atlassian-centric perspective on tooling, take a look at - this post on how to - [choose the right DevOps tools](http://blogs.atlassian.com/2016/03/how-to-choose-devops-tools/) + this guide on how to + [choose the right DevOps tools](https://www.atlassian.com/devops/devops-tools) which is biased towards their tools but still has some good insight such as using automated testing to provide immediate awareness of defects that require fixing. @@ -47,10 +47,10 @@ The following resources give advice and approaches for building the right teams, culture, processes and tools into software development organizations. * [DevOps vs. Platform Engineering](https://alexgaynor.net/2015/mar/06/devops-vs-platform-engineering/) - considers DevOps an ad hoc approach to developing software while building - a platform is a strict contract. I see this as "DevOps is a process", - while a "platform is code". Running code is better than any organizational - process. + considers DevOps to be an ad hoc approach to developing software while + building a platform is a strict contract. I see this as "DevOps is a + process", while a "platform is code". Running code is better than any + organizational process. * The open source [PagerDuty Incident Response guide](https://response.pagerduty.com/) is the @@ -58,6 +58,12 @@ teams, culture, processes and tools into software development organizations. their services running and putting them out for other developers to consume. Highly recommended. +* [Introduction to DevOps and Software Delivery Performance](https://www.stridenyc.com/blog/devops-and-software-delivery-performance) + explains the four key delivery metrics of Delivery Lead Time, + Deployment Frequency, Time to Restore Service, and Change Fail Rate, + and then gives a high-level overview of technical, process and + cultural capabilities that impact these metrics. + * [Operations for software developers for beginners](https://jvns.ca/blog/2016/10/15/operations-for-software-developers-for-beginners/) gives advice to developers who have never done operations work and been on call for outages before in their career. The advantage of DevOps @@ -82,10 +88,6 @@ teams, culture, processes and tools into software development organizations. your pager goes off, ownership and how startups can be different from large companies with their incident responses. -* [Bing: Continuous Delivery](http://stories.visualstudio.com/bing-continuous-delivery/) - is an impressive visual story that explains the practices for how their - team delivers updates to the search engine. - * [Why are we racing to DevOps?](http://www.cio.com/article/3015237/application-development/why-are-we-racing-to-devops.html) is a very high level summary of the benefits of DevOps to IT organizations. It's not specific to Python and doesn't dive into the details, but it's diff --git a/content/pages/06-devops/01-monitoring.markdown b/content/pages/06-devops/01-monitoring.markdown index 95efe2cf4..50021eecf 100644 --- a/content/pages/06-devops/01-monitoring.markdown +++ b/content/pages/06-devops/01-monitoring.markdown @@ -112,7 +112,7 @@ Application Performance Monitoring (APM) * [New Relic](http://newrelic.com/) provides application and database monitoring as well as plug ins for capturing and analyzing data about - other devleoper tools in your stack, such as [Twilio](/twilio.html). + other developer tools in your stack, such as [Twilio](/twilio.html). * [Opbeat](https://opbeat.com) Built for django. Opbeat combines performance metrics, release tracking, and error logging into a single simple service. * [Scout](https://scoutapp.com/python-monitoring) monitors the performance of Django and Flask apps, auto-instrumenting views, SQL queries, templates, and more. @@ -146,11 +146,6 @@ Incident Management * [The Virtues of Monitoring](http://www.paperplanes.de/2011/1/5/the_virtues_of_monitoring.html) -* [Effortless Monitoring with collectd, Graphite, and Docker](http://blog.docker.io/2013/07/effortless-monitoring-with-collectd-graphite-and-docker/) - -* [Practical Guide to StatsD/Graphite Monitoring](http://matt.aimonetti.net/posts/2013/06/26/practical-guide-to-graphite-monitoring/) - is a detailed guide with code examples for monitoring infrastructure. - * Bit.ly describes the "[10 Things They Forgot to Monitor](http://word.bitly.com/post/74839060954/ten-things-to-monitor)" beyond the standard metrics such as disk & memory usage. diff --git a/content/pages/06-devops/06-web-app-performance.markdown b/content/pages/06-devops/06-web-app-performance.markdown index faa195727..8a954546f 100644 --- a/content/pages/06-devops/06-web-app-performance.markdown +++ b/content/pages/06-devops/06-web-app-performance.markdown @@ -49,10 +49,6 @@ database queries, page size and many other factors. is a 20 minute code-first demo that shows how to get a realistic estimate for how many requests per second your web application will be able to handle. -* [How to Interpret Site Performance Tests](https://fly.io/articles/how-to-understand-performance-tests/) - covers the difference between client, page and connection speed tests - as well as a bit on caching performance. - * [Practical scaling techniques for websites](https://hackernoon.com/practical-scaling-techniques-for-web-sites-554a38dbd492) examines how to improve your website performance with asynchronous [task queues](/task-queues.html), [database](/databases.html) optimization diff --git a/content/pages/06-devops/11-caching.markdown b/content/pages/06-devops/11-caching.markdown index 5a8e94abf..c8aebbbf1 100644 --- a/content/pages/06-devops/11-caching.markdown +++ b/content/pages/06-devops/11-caching.markdown @@ -45,19 +45,10 @@ A cache can be created for multiple layers of the stack. reading even though the author is describing his Microsoft code as the impetus for writing the content. -* While caching is a useful technique in many situations, it's important - to also note that there are - [downsides to caching](https://msol.io/blog/tech/2015/09/05/youre-probably-wrong-about-caching/) - that many developers fail to take into consideration. - * [Caching at Reddit](https://redditblog.com/2017/1/17/caching-at-reddit/) covers monitoring, tuning and scaling for the very high scale [Reddit.com](https://www.reddit.com/) website. -* [Mastering HTTP caching](https://blog.fortrabbit.com/mastering-http-caching) - provides more advanced advice on caching dynamic as well as static - content via CDNs and other configurations. - ## Caching learning checklist diff --git a/content/pages/06-devops/14-logging.markdown b/content/pages/06-devops/14-logging.markdown index 3fa5b0e30..9738d98fc 100644 --- a/content/pages/06-devops/14-logging.markdown +++ b/content/pages/06-devops/14-logging.markdown @@ -55,8 +55,9 @@ certain threshold. There are libraries for most major languages, including python. Saves data in Elasticache. -* [Logstash](http://logstash.net/) Similar to Graylog2, logstash offers - features to programmatically configure log data workflows. +* [Logstash](https://www.elastic.co/guide/en/logstash/current/index.html). + Similar to Graylog2, logstash offers features to programmatically + configure log data workflows. * [Scribe](https://github.com/facebook/scribe) A project written by Facebook to aggregate logs. It's designed to run on multiple servers and scale with @@ -112,7 +113,7 @@ certain threshold. * [Good logging practice in Python](http://victorlin.me/posts/2012/08/26/good-logging-practice-in-python) shows how to use the standard library to log data from your application. Definitely worth a read as most applications do not log nearly enough - output to help debuggin when things go wrong, or to determine if something + output to help debugging when things go wrong, or to determine if something is going wrong. * [Structured Logging: The Best Friend You’ll Want When Things Go Wrong](https://engineering.grab.com/structured-logging) diff --git a/content/pages/06-devops/18-web-analytics.markdown b/content/pages/06-devops/18-web-analytics.markdown index 28c0a58a9..6430b1057 100644 --- a/content/pages/06-devops/18-web-analytics.markdown +++ b/content/pages/06-devops/18-web-analytics.markdown @@ -38,9 +38,10 @@ application before taking some action, such as purchasing your service. ## Open source web analytics projects -* [Piwik](http://piwik.org/) is a web analytics platform you can host yourself. - Piwik is a solid choice if you cannot use Google Analytics or want to - customize your own web analytics platform. +* [Matoma](https://matomo.org/) (formerly Piwik), is a web analytics + platform that you can host yourself. Matoma is a solid choice if you + cannot use Google Analytics or want to customize your own web analytics + software. * [Shynet](https://github.com/milesmcc/shynet) is a lightweight, privacy-friendly cookie-free web analytics application written in Python. @@ -109,10 +110,6 @@ application before taking some action, such as purchasing your service. is not for everyone but it is really useful if you want to avoid the Google data trap. -* This beginner's guide to - [math and stats behind web analytics](http://www.seotakeaways.com/beginners-guide-maths-stats-web-analytics/) - provides some context for understanding and reasoning about web traffic. - * [An Analytics Primer for Developers](https://hacks.mozilla.org/2015/03/an-analytics-primer-for-developers/) by Mozilla explains what to track, choosing an analytics platform and how to serve up the analytics JavaScript asynchronously. @@ -142,8 +139,8 @@ application before taking some action, such as purchasing your service. ## Web analytics learning checklist -1. Add Google Analytics or Piwik to your application. Both are free and while - Piwik is not as powerful as Google Analytics you can self-host the +1. Add Google Analytics or Matoma to your application. Both are free and while + Matoma is not as powerful as Google Analytics you can self-host the application which is the only option in many environments. 1. Think critically about the factors that will make your application diff --git a/content/pages/10-working/00-gpt-3.markdown b/content/pages/10-working/00-gpt-3.markdown index 1e54ef4ea..04e1726a5 100644 --- a/content/pages/10-working/00-gpt-3.markdown +++ b/content/pages/10-working/00-gpt-3.markdown @@ -107,6 +107,11 @@ These resources range from broad philosophy of what GPT-3 means for machine learning to specific technical details for how the model is trained. +* [The Ultimate Guide to OpenAI's GPT-3 Language Model](https://www.twilio.com/blog/ultimate-guide-openai-gpt-3-language-model) + is a detailed tutorial on how to use OpenAI's playground user interface, + what the parameters do, and how to convert what you have done in + the playground into a Python script that calls their API. + * [OpenAI's GPT-3 Language Model: A Technical Overview](https://lambdalabs.com/blog/demystifying-gpt-3/) and [GPT-3: A Hitchhiker's Guide](https://lambdalabs.com/blog/gpt-3/) @@ -115,15 +120,15 @@ is trained. researchers on its usage, and some initial resources to get a better understanding of what this model is capable of performing. +* [What Is GPT-3: How It Works and Why You Should Care](https://www.twilio.com/blog/what-is-gpt-3) + presents a high-level accessible overview of GPT-3, how it compares + to other language models, and resources to learn more. + * [How GPT3 Works - Visualizations and Animations](https://jalammar.github.io/how-gpt3-works-visualizations-animations/) contains some wonderful animated visuals to show how the model is trained and what happens in various scenarios such as text output and code generation. -* [Building a Chatbot with OpenAI's GPT-3 engine, Twilio SMS and Python](https://www.twilio.com/blog/openai-gpt-3-chatbot-python-twilio-sms) - is a step-by-step tutorial for using GPT-3 as a smart backend - for an SMS-based chatbot powered by the [Twilio API](/twilio.html). - * [GPT 3 Demo and Explanation](https://www.youtube.com/watch?v=8psgEDhT1MM) is a video that gives a brief overview of GPT-3 and shows a bunch of live demos for what has so far been created with this technology. @@ -132,11 +137,35 @@ is trained. points out that many of the good examples on social media have been cherry picked to impress readers. -* [gpt-3-experiments](https://github.com/minimaxir/gpt-3-experiments) - contains Python code open sourced under the MIT license that - shows how to interact with the API. - * [Why GPT-3 matters](https://leogao.dev/2020/05/29/GPT-3-A-Brief-Summary/) compares and contrasts this model with similar models that have been developed and tries to give an overview of where each one stands with its strengths and weaknesses. + + +## GPT-3 tutorials +* [Building a Chatbot with OpenAI's GPT-3 engine, Twilio SMS and Python](https://www.twilio.com/blog/openai-gpt-3-chatbot-python-twilio-sms) + is a step-by-step tutorial for using GPT-3 as a smart backend + for an SMS-based chatbot powered by the [Twilio API](/twilio.html). + +* [Automating my job by using GPT-3 to generate database-ready SQL to answer business questions](https://blog.seekwell.io/gpt3) + walks through how the author created a bridge to translate between + plain English-language questions and + [relational database](/databases.html) SQL. The post provides both + a story for why someone would want to use GPT-3 for this purpose and + incremental steps for how the author started and figured out how + to make it better. In the end it does not quite work in all scenarios + but the proof of concept is impressive and the story is a fun read. + +* [gpt-3-experiments](https://github.com/minimaxir/gpt-3-experiments) + contains Python code open sourced under the MIT license that + shows how to interact with the API. + +* [Twilio](/twilio.html) put out a series of fun GPT-3 tutorials that show + the range of creative outputs the model can generate: + + * [Control a Spooky Ghost Writer for Halloween with OpenAI's GPT-3 Engine, Python, and Twilio WhatsApp API](https://www.twilio.com/blog/ghost-writer-spooky-openai-gpt3-python-whatsapp) + * [Generating Lyrics in the Style of your Favorite Artist with Python, OpenAI's GPT-3 and Twilio SMS](https://www.twilio.com/blog/generating-lyrics-in-the-style-of-your-favorite-artist-with-python-openai-s-gpt-3-and-twilio-sms) + * [Automated Yugioh Deckbuilding in Python with OpenAI's GPT-3 and Twilio SMS](https://www.twilio.com/blog/building-computer-generated-yugioh-decks-in-python-with-openai-s-gpt-3-and-twilio-sms) + * [Build a Telephone Chatbot with GPT-3 and Twilio Autopilot](https://www.twilio.com/blog/build-telephone-chatbot-gpt3-twilio-autopilot) + diff --git a/content/pages/10-working/01-event-streams.markdown b/content/pages/10-working/01-event-streams.markdown new file mode 100644 index 000000000..0daa6e45f --- /dev/null +++ b/content/pages/10-working/01-event-streams.markdown @@ -0,0 +1,58 @@ +title: Event Streams +category: page +slug: event-streams +sortorder: 1002 +toc: False +sidebartitle: Event Streams +meta: An event stream is a log of one or more events. + + +Event streams are a log of one or more "things that happen", which are +usually referred to as events. Event streams are +conceptually focused around events than objects or tables, which are +the typical storage unit of [relational databases](/databases.html). + +Apache Kafka and Gazette are a popular open source implementations of event +streams. Amazon Web Services' Kinesis and +[Azure Event-Hubs](https://azure.microsoft.com/en-us/services/event-hubs/) +are proprietary hosted implementations. + + +## Why do event streams matter to developers? +The way that data is stored affects how you can work with it. Constraints +and guarantees like consistency make it easier to code certain applications +but harder to build other types of applications that need performance in +different ways. Event streams make it easier to build applications that +analyze large amounts of constantly-updated data because the events are +not stored relationally. + + +## How are event streams typically stored? +Some applications, such as aggregating millions of sensors, or thousands +of streaming cameras, constantly output large amounts of data with no breaks. +It is difficult to process such large volumes of data in traditional data +stores, so event streams are built off of a simpler data structure: logs. + +Logs are ordered sequences of events and they typically have less constraints +than a database. In event streams, logs are also immutable. They do not change +once they are written. Instead, newer events are written in the sequence as +state changes. The reduced constraints compared to a database and the +immutability mean that logs can handle the high throughput writes needed to +keep up with the constant flood of data from the source of an event stream. + + +## Event stream resources +* [What is Apache Kafka?](https://www.youtube.com/watch?v=FKgi3n-FyNU) sounds + like it just focuses on Kafka but it actually covers the fundamental + concepts behind event streams and how they fit into + [microservices](/microservices.html) architectures. + +* Quora has a solid answer to the question of + [what is an event stream?](https://www.quora.com/What-is-an-event-stream). + +* [Summary of the Amazon Kinesis Event in the Northern Virginia (US-EAST-1) Region](https://aws.amazon.com/message/11201/) + is specific to AWS Kinesis but it explains how Amazon uses event + streams at scale to run and coordinate a significant number of their + services. When their event streams service went down... it took a + whole lot of other stuff down at the same time. There is also some + [additional analysis in this post by an independent developer](https://ryanfrantz.com/posts/aws-kinesis-outage-analysis.html). diff --git a/content/pages/10-working/18-developer-demos.md b/content/pages/10-working/18-developer-demos.md new file mode 100644 index 000000000..a195edfdc --- /dev/null +++ b/content/pages/10-working/18-developer-demos.md @@ -0,0 +1,29 @@ +title: Demoing to Software Developers +category: page +slug: demo-software-developers +sortorder: 1018 +toc: False +sidebartitle: Developer Demos +meta: How to give technical demos to software developers. + + +Creating and executing an appealing technical demo to an audience of software +developers is a ton of work, but there is no better way to get people +legitimately interested in your product if you land a great demo with +an appropriate audience. The inherent difficulty involved in exceptional +technical demos also creates a large barrier that prevent others from simply +copying your work, which can happen with technical blog content. + +To achieve a standout technical demo you must: + +1. show how to easily solve a difficult technical problem +1. speak plainly but accurately +1. do it live, including writing code if required +1. tell a story with a narrative arc +1. rehearse constantly, both the happy path and recovering from errors + + +Examples: +* [Twilio Phone Calls Demo at NY Tech Meetup (2010)](https://www.youtube.com/watch?v=-VuXIgp9S7o) +* [Concurrency from the Ground Up (2015)](https://www.youtube.com/watch?v=MCs5OvhV9S4) +* [The Mother of All Demos (1968)](https://www.youtube.com/watch?v=yJDv-zdhzMY) diff --git a/content/pages/examples/django/django-extensions-plug-ins.markdown b/content/pages/examples/django/django-extensions-plug-ins.markdown index 626e23cc5..632cde7da 100644 --- a/content/pages/examples/django/django-extensions-plug-ins.markdown +++ b/content/pages/examples/django/django-extensions-plug-ins.markdown @@ -75,6 +75,18 @@ Code from django-angular is shown on: * [django.utils.html format_html](/django-utils-html-format-html-examples.html) * [django.urls.exceptions NoReverseMatch](/django-urls-exceptions-noreversematch-examples.html) +### django-appmail +[Django-Appmail](https://github.com/yunojuno/django-appmail) +([PyPI package information](https://pypi.org/project/django-appmail/)) +is a [Django](/django.html) app for handling transactional email templates. +While the project began development as a way to work with the Mandrill +transactional [API](/application-programming-interfaces.html), it is +not exclusive to that API. The project simply provides a way to store +and render email content. The library does not send or receive emails. + +Django-Appmail is open sourced under the +[MIT license](https://github.com/yunojuno/django-appmail/blob/master/LICENSE). + ### django-axes [django-axes](https://github.com/jazzband/django-axes/) @@ -321,6 +333,18 @@ The django-jsonfield project is open source under the [MIT license](https://github.com/dmkoch/django-jsonfield/blob/master/LICENSE). +### django-linear-migrations +[django-linear-migrations](https://github.com/adamchainz/django-linear-migrations) +([PyPI package information](https://pypi.org/project/django-linear-migrations/)) +is a [Django](/django.html) code library to mitigate conflicting database +migrations, which can cause non-deterministic behavior in different +environments. The +[introductory blog post by the package author](https://adamj.eu/tech/2020/12/10/introducing-django-linear-migrations/) +does a good job of explaining the problem and how this library prevents +the issue. This library is open sourced under the +[MIT license](https://github.com/adamchainz/django-linear-migrations/blob/master/LICENSE). + + ### django-loginas [django-loginas](https://github.com/skorokithakis/django-loginas) ([PyPI package information](https://pypi.org/project/django-loginas/)) @@ -470,6 +494,18 @@ The project is open sourced under the [Encode OSS Ltd. license](https://github.com/encode/django-rest-framework/blob/master/LICENSE.md). +### Django Request Token +[Django Request Token](https://github.com/yunojuno/django-request-token) +([PyPI package information](https://pypi.org/project/django-request-token/0.13/)) +encapsulates the logic for issuing expiring and one-time tokens +with a [Django](/django.html) web application to use with protected URLs. +Note that [PostgreSQL](/postgresql.html) as your backend +[database](/databases.html) is a dependency for using this project. + +The Django Request Token project is open sourced under the +[MIT license](https://github.com/yunojuno/django-request-token/blob/master/LICENSE). + + ### django-rq [django-rq](https://github.com/rq/django-rq) ([PyPI package information](https://pypi.org/project/django-rq/)) @@ -568,6 +604,23 @@ under the [MIT license](https://github.com/yunojuno/django-user-visit/blob/master/LICENSE). +### django-version-checks +[django-version-checks](https://github.com/adamchainz/django-version-checks) +([PyPI package](https://pypi.org/project/django-version-checks/)) +is a code library to ensure external system dependencies match +desired versions. For example, a specific version of +[PostgreSQL](/postgresql.html) or [MySQL](/mysql.html) as your database +backend. This is different from using `pip` and a `requirements.txt` file, +because those are Python dependencies, rather than system-wide software. +The +[introductory blog post](https://adamj.eu/tech/2020/12/14/introducing-django-version-checks/) +for the project has some good reasons why these external dependencies +can cause problems if they vary from the expected versions. + +django-version-checks is provided as open source under the +[MIT license](https://github.com/adamchainz/django-version-checks/blob/master/LICENSE). + + ### django-webshell [django-webshell](https://github.com/onrik/django-webshell) is an extension for executing arbitrary code in the diff --git a/content/pages/examples/django/django-template-base-filterexpression.markdown b/content/pages/examples/django/django-template-base-filterexpression.markdown index aa466b309..6a5d325a2 100644 --- a/content/pages/examples/django/django-template-base-filterexpression.markdown +++ b/content/pages/examples/django/django-template-base-filterexpression.markdown @@ -1,14 +1,27 @@ title: django.template.base FilterExpression Example Code category: page slug: django-template-base-filterexpression-examples -sortorder: 500011364 +sortorder: 500011370 toc: False sidebartitle: django.template.base FilterExpression -meta: Python example code for the FilterExpression class from the django.template.base module of the Django project. - - -FilterExpression is a class within the django.template.base module of the Django project. - +meta: Example code for understanding how to use the FilterExpression class from the django.template.base module of the Django project. + + +`FilterExpression` is a class within the `django.template.base` module of the Django project. + +Context, +Node, +NodeList, +Parser, +Template, +TemplateSyntaxError, +TextNode, +Token, +TokenType, +VariableDoesNotExist, +VariableNode, +and token_kwargs +are several other callables with code examples from the same `django.template.base` package. ## Example 1 from django-sitetree [django-sitetree](https://github.com/idlesign/django-sitetree) diff --git a/content/pages/examples/django/django-template-base-node.markdown b/content/pages/examples/django/django-template-base-node.markdown index a7557b13a..d12b75a6f 100644 --- a/content/pages/examples/django/django-template-base-node.markdown +++ b/content/pages/examples/django/django-template-base-node.markdown @@ -1,14 +1,27 @@ title: django.template.base Node Example Code category: page slug: django-template-base-node-examples -sortorder: 500011365 +sortorder: 500011371 toc: False sidebartitle: django.template.base Node -meta: Python example code for the Node class from the django.template.base module of the Django project. - - -Node is a class within the django.template.base module of the Django project. - +meta: Example code for understanding how to use the Node class from the django.template.base module of the Django project. + + +`Node` is a class within the `django.template.base` module of the Django project. + +Context, +FilterExpression, +NodeList, +Parser, +Template, +TemplateSyntaxError, +TextNode, +Token, +TokenType, +VariableDoesNotExist, +VariableNode, +and token_kwargs +are several other callables with code examples from the same `django.template.base` package. ## Example 1 from django-angular [django-angular](https://github.com/jrief/django-angular) diff --git a/content/pages/examples/django/django-template-base-nodelist.markdown b/content/pages/examples/django/django-template-base-nodelist.markdown index c178a8121..82615eb0b 100644 --- a/content/pages/examples/django/django-template-base-nodelist.markdown +++ b/content/pages/examples/django/django-template-base-nodelist.markdown @@ -1,14 +1,27 @@ title: django.template.base NodeList Example Code category: page slug: django-template-base-nodelist-examples -sortorder: 500011366 +sortorder: 500011372 toc: False sidebartitle: django.template.base NodeList -meta: Python example code for the NodeList class from the django.template.base module of the Django project. - - -NodeList is a class within the django.template.base module of the Django project. - +meta: Example code for understanding how to use the NodeList class from the django.template.base module of the Django project. + + +`NodeList` is a class within the `django.template.base` module of the Django project. + +Context, +FilterExpression, +Node, +Parser, +Template, +TemplateSyntaxError, +TextNode, +Token, +TokenType, +VariableDoesNotExist, +VariableNode, +and token_kwargs +are several other callables with code examples from the same `django.template.base` package. ## Example 1 from django-angular [django-angular](https://github.com/jrief/django-angular) diff --git a/content/pages/examples/django/django-template-base-parser.markdown b/content/pages/examples/django/django-template-base-parser.markdown index a3e3fdbe2..d935c982b 100644 --- a/content/pages/examples/django/django-template-base-parser.markdown +++ b/content/pages/examples/django/django-template-base-parser.markdown @@ -1,14 +1,27 @@ title: django.template.base Parser Example Code category: page slug: django-template-base-parser-examples -sortorder: 500011367 +sortorder: 500011373 toc: False sidebartitle: django.template.base Parser -meta: Python example code for the Parser class from the django.template.base module of the Django project. +meta: Example code for understanding how to use the Parser class from the django.template.base module of the Django project. -Parser is a class within the django.template.base module of the Django project. +`Parser` is a class within the `django.template.base` module of the Django project. +Context, +FilterExpression, +Node, +NodeList, +Template, +TemplateSyntaxError, +TextNode, +Token, +TokenType, +VariableDoesNotExist, +VariableNode, +and token_kwargs +are several other callables with code examples from the same `django.template.base` package. ## Example 1 from django-sitetree [django-sitetree](https://github.com/idlesign/django-sitetree) @@ -84,7 +97,7 @@ class TreeItemChoiceField(ChoiceField): context.update({'request': object()}) choices_str = sitetree_tree( -~~ Parser(None), Token(token_type=TOKEN_BLOCK, contents=tree_token) +~~ Parser([]), Token(token_type=TOKEN_BLOCK, contents=tree_token) ).render(context) tree_choices = [(ITEMS_FIELD_ROOT_ID, self.root_title)] diff --git a/content/pages/examples/django/django-template-base-template.markdown b/content/pages/examples/django/django-template-base-template.markdown index 70cb7dca7..23d44a910 100644 --- a/content/pages/examples/django/django-template-base-template.markdown +++ b/content/pages/examples/django/django-template-base-template.markdown @@ -1,14 +1,27 @@ title: django.template.base Template Example Code category: page slug: django-template-base-template-examples -sortorder: 500011368 +sortorder: 500011374 toc: False sidebartitle: django.template.base Template -meta: Python example code for the Template class from the django.template.base module of the Django project. +meta: Example code for understanding how to use the Template class from the django.template.base module of the Django project. -Template is a class within the django.template.base module of the Django project. +`Template` is a class within the `django.template.base` module of the Django project. +Context, +FilterExpression, +Node, +NodeList, +Parser, +TemplateSyntaxError, +TextNode, +Token, +TokenType, +VariableDoesNotExist, +VariableNode, +and token_kwargs +are several other callables with code examples from the same `django.template.base` package. ## Example 1 from django-cms [django-cms](https://github.com/divio/django-cms) @@ -27,8 +40,6 @@ from django.contrib.admin.sites import site from django.template import Context ~~from django.template.base import Template -from six import text_type - from cms.api import add_plugin from cms.models import StaticPlaceholder, Placeholder, UserSettings from cms.tests.test_plugins import PluginsTestBaseCase diff --git a/content/pages/examples/django/django-template-base-templatesyntaxerror.markdown b/content/pages/examples/django/django-template-base-templatesyntaxerror.markdown index 4dd549d7c..9050c318a 100644 --- a/content/pages/examples/django/django-template-base-templatesyntaxerror.markdown +++ b/content/pages/examples/django/django-template-base-templatesyntaxerror.markdown @@ -1,14 +1,27 @@ title: django.template.base TemplateSyntaxError Example Code category: page slug: django-template-base-templatesyntaxerror-examples -sortorder: 500011369 +sortorder: 500011375 toc: False sidebartitle: django.template.base TemplateSyntaxError -meta: Python example code for the TemplateSyntaxError class from the django.template.base module of the Django project. - - -TemplateSyntaxError is a class within the django.template.base module of the Django project. - +meta: Example code for understanding how to use the TemplateSyntaxError class from the django.template.base module of the Django project. + + +`TemplateSyntaxError` is a class within the `django.template.base` module of the Django project. + +Context, +FilterExpression, +Node, +NodeList, +Parser, +Template, +TextNode, +Token, +TokenType, +VariableDoesNotExist, +VariableNode, +and token_kwargs +are several other callables with code examples from the same `django.template.base` package. ## Example 1 from django-sitetree [django-sitetree](https://github.com/idlesign/django-sitetree) diff --git a/content/pages/examples/django/django-template-base-textnode.markdown b/content/pages/examples/django/django-template-base-textnode.markdown index e47a68846..ba9d0fd65 100644 --- a/content/pages/examples/django/django-template-base-textnode.markdown +++ b/content/pages/examples/django/django-template-base-textnode.markdown @@ -1,14 +1,27 @@ title: django.template.base TextNode Example Code category: page slug: django-template-base-textnode-examples -sortorder: 500011370 +sortorder: 500011376 toc: False sidebartitle: django.template.base TextNode -meta: Python example code for the TextNode class from the django.template.base module of the Django project. - - -TextNode is a class within the django.template.base module of the Django project. - +meta: Example code for understanding how to use the TextNode class from the django.template.base module of the Django project. + + +`TextNode` is a class within the `django.template.base` module of the Django project. + +Context, +FilterExpression, +Node, +NodeList, +Parser, +Template, +TemplateSyntaxError, +Token, +TokenType, +VariableDoesNotExist, +VariableNode, +and token_kwargs +are several other callables with code examples from the same `django.template.base` package. ## Example 1 from django-angular [django-angular](https://github.com/jrief/django-angular) diff --git a/content/pages/examples/django/django-template-base-token-kwargs.markdown b/content/pages/examples/django/django-template-base-token-kwargs.markdown index ecafe2225..0daeda14c 100644 --- a/content/pages/examples/django/django-template-base-token-kwargs.markdown +++ b/content/pages/examples/django/django-template-base-token-kwargs.markdown @@ -1,14 +1,27 @@ title: django.template.base token_kwargs Example Code category: page slug: django-template-base-token-kwargs-examples -sortorder: 500011375 +sortorder: 500011381 toc: False sidebartitle: django.template.base token_kwargs -meta: Python example code for the token_kwargs callable from the django.template.base module of the Django project. - - -token_kwargs is a callable within the django.template.base module of the Django project. - +meta: Python example code that shows how to use the token_kwargs callable from the django.template.base module of the Django project. + + +`token_kwargs` is a callable within the `django.template.base` module of the Django project. + +Context, +FilterExpression, +Node, +NodeList, +Parser, +Template, +TemplateSyntaxError, +TextNode, +Token, +TokenType, +VariableDoesNotExist, +and VariableNode +are several other callables with code examples from the same `django.template.base` package. ## Example 1 from django-floppyforms [django-floppyforms](https://github.com/jazzband/django-floppyforms) diff --git a/content/pages/examples/django/django-template-base-token.markdown b/content/pages/examples/django/django-template-base-token.markdown index 6c9da8eaa..5610cf6fe 100644 --- a/content/pages/examples/django/django-template-base-token.markdown +++ b/content/pages/examples/django/django-template-base-token.markdown @@ -1,14 +1,27 @@ title: django.template.base Token Example Code category: page slug: django-template-base-token-examples -sortorder: 500011371 +sortorder: 500011377 toc: False sidebartitle: django.template.base Token -meta: Python example code for the Token class from the django.template.base module of the Django project. +meta: Example code for understanding how to use the Token class from the django.template.base module of the Django project. -Token is a class within the django.template.base module of the Django project. +`Token` is a class within the `django.template.base` module of the Django project. +Context, +FilterExpression, +Node, +NodeList, +Parser, +Template, +TemplateSyntaxError, +TextNode, +TokenType, +VariableDoesNotExist, +VariableNode, +and token_kwargs +are several other callables with code examples from the same `django.template.base` package. ## Example 1 from django-sitetree [django-sitetree](https://github.com/idlesign/django-sitetree) @@ -84,7 +97,7 @@ class TreeItemChoiceField(ChoiceField): context.update({'request': object()}) choices_str = sitetree_tree( -~~ Parser(None), Token(token_type=TOKEN_BLOCK, contents=tree_token) +~~ Parser([]), Token(token_type=TOKEN_BLOCK, contents=tree_token) ).render(context) tree_choices = [(ITEMS_FIELD_ROOT_ID, self.root_title)] diff --git a/content/pages/examples/django/django-template-base-tokentype.markdown b/content/pages/examples/django/django-template-base-tokentype.markdown index ff8af3738..6064b392d 100644 --- a/content/pages/examples/django/django-template-base-tokentype.markdown +++ b/content/pages/examples/django/django-template-base-tokentype.markdown @@ -1,14 +1,27 @@ title: django.template.base TokenType Example Code category: page slug: django-template-base-tokentype-examples -sortorder: 500011372 +sortorder: 500011378 toc: False sidebartitle: django.template.base TokenType -meta: Python example code for the TokenType class from the django.template.base module of the Django project. - - -TokenType is a class within the django.template.base module of the Django project. - +meta: Example code for understanding how to use the TokenType class from the django.template.base module of the Django project. + + +`TokenType` is a class within the `django.template.base` module of the Django project. + +Context, +FilterExpression, +Node, +NodeList, +Parser, +Template, +TemplateSyntaxError, +TextNode, +Token, +VariableDoesNotExist, +VariableNode, +and token_kwargs +are several other callables with code examples from the same `django.template.base` package. ## Example 1 from django-sitetree [django-sitetree](https://github.com/idlesign/django-sitetree) diff --git a/content/pages/examples/django/django-template-base-variabledoesnotexist.markdown b/content/pages/examples/django/django-template-base-variabledoesnotexist.markdown index cd6ea6d80..a1894a291 100644 --- a/content/pages/examples/django/django-template-base-variabledoesnotexist.markdown +++ b/content/pages/examples/django/django-template-base-variabledoesnotexist.markdown @@ -1,14 +1,27 @@ title: django.template.base VariableDoesNotExist Example Code category: page slug: django-template-base-variabledoesnotexist-examples -sortorder: 500011373 +sortorder: 500011379 toc: False sidebartitle: django.template.base VariableDoesNotExist -meta: Python example code for the VariableDoesNotExist class from the django.template.base module of the Django project. +meta: Example code for understanding how to use the VariableDoesNotExist class from the django.template.base module of the Django project. -VariableDoesNotExist is a class within the django.template.base module of the Django project. +`VariableDoesNotExist` is a class within the `django.template.base` module of the Django project. +Context, +FilterExpression, +Node, +NodeList, +Parser, +Template, +TemplateSyntaxError, +TextNode, +Token, +TokenType, +VariableNode, +and token_kwargs +are several other callables with code examples from the same `django.template.base` package. ## Example 1 from django-pipeline [django-pipeline](https://github.com/jazzband/django-pipeline) @@ -34,7 +47,7 @@ import subprocess from django.contrib.staticfiles.storage import staticfiles_storage from django import template -~~from django.template.base import Context, VariableDoesNotExist +~~from django.template.base import VariableDoesNotExist from django.template.loader import render_to_string from django.utils.safestring import mark_safe diff --git a/content/pages/examples/django/django-template-base-variablenode.markdown b/content/pages/examples/django/django-template-base-variablenode.markdown index 8a0e361a0..a2d26decd 100644 --- a/content/pages/examples/django/django-template-base-variablenode.markdown +++ b/content/pages/examples/django/django-template-base-variablenode.markdown @@ -1,14 +1,27 @@ title: django.template.base VariableNode Example Code category: page slug: django-template-base-variablenode-examples -sortorder: 500011374 +sortorder: 500011380 toc: False sidebartitle: django.template.base VariableNode -meta: Python example code for the VariableNode class from the django.template.base module of the Django project. +meta: Example code for understanding how to use the VariableNode class from the django.template.base module of the Django project. -VariableNode is a class within the django.template.base module of the Django project. +`VariableNode` is a class within the `django.template.base` module of the Django project. +Context, +FilterExpression, +Node, +NodeList, +Parser, +Template, +TemplateSyntaxError, +TextNode, +Token, +TokenType, +VariableDoesNotExist, +and token_kwargs +are several other callables with code examples from the same `django.template.base` package. ## Example 1 from django-angular [django-angular](https://github.com/jrief/django-angular) @@ -126,8 +139,6 @@ from django.template import TemplateSyntaxError, NodeList, Variable, Context, Te from django.template.loader import get_template from django.template.loader_tags import BlockNode, ExtendsNode, IncludeNode -from six import string_types - from sekizai.helpers import get_varname from cms.exceptions import DuplicatePlaceholderWarning @@ -147,6 +158,8 @@ def get_context(): context.template = Template('') return context else: + return {} + ## ... source file continues with no further VariableNode examples... diff --git a/content/pages/examples/django/django-template-context-context.markdown b/content/pages/examples/django/django-template-context-context.markdown new file mode 100644 index 000000000..5b3926fb2 --- /dev/null +++ b/content/pages/examples/django/django-template-context-context.markdown @@ -0,0 +1,186 @@ +title: django.template.context Context Example Code +category: page +slug: django-template-context-context-examples +sortorder: 500011382 +toc: False +sidebartitle: django.template.context Context +meta: Example code for understanding how to use the Context class from the django.template.context module of the Django project. + + +`Context` is a class within the `django.template.context` module of the Django project. + + + +## Example 1 from django-cms +[django-cms](https://github.com/divio/django-cms) +([project website](https://www.django-cms.org/en/)) is a Python-based +content management system (CMS) [library](https://pypi.org/project/django-cms/) +for use with Django web apps that is open sourced under the +[BSD 3-Clause "New"](https://github.com/divio/django-cms/blob/develop/LICENSE) +license. + +[**django-cms / cms / test_utils / testcases.py**](https://github.com/divio/django-cms/blob/develop/cms/test_utils/testcases.py) + +```python +# testcases.py +import json +import sys +import warnings + +from urllib.parse import unquote, urljoin + +from django.conf import settings +from django.contrib.auth import get_user_model +from django.contrib.auth.models import AnonymousUser, Permission +from django.contrib.sites.models import Site +from django.core.cache import cache +from django.core.exceptions import ObjectDoesNotExist +from django.forms.models import model_to_dict +from django.template import engines +~~from django.template.context import Context +from django.test import testcases +from django.test.client import RequestFactory +from django.urls import reverse +from django.utils.http import urlencode +from django.utils.timezone import now +from django.utils.translation import activate +from menus.menu_pool import menu_pool + +from cms.api import create_page +from cms.constants import ( + PUBLISHER_STATE_DEFAULT, + PUBLISHER_STATE_DIRTY, + PUBLISHER_STATE_PENDING, +) +from cms.plugin_rendering import ContentRenderer, StructureRenderer +from cms.models import Page +from cms.models.permissionmodels import ( + GlobalPagePermission, + PagePermission, + PageUser, +) +from cms.test_utils.util.context_managers import UserLoginContext +from cms.utils.conf import get_cms_setting +from cms.utils.permissions import set_current_user + + +## ... source file abbreviated to get to Context examples ... + + + + def create_homepage(self, *args, **kwargs): + homepage = create_page(*args, **kwargs) + homepage.set_as_homepage() + return homepage.reload() + + def move_page(self, page, target_page, position="first-child"): + page.move_page(target_page.node, position) + return self.reload_page(page) + + def reload_page(self, page): + return self.reload(page) + + def reload(self, obj): + return obj.__class__.objects.get(pk=obj.pk) + + def get_pages_root(self): + return unquote(reverse("pages-root")) + + def get_context(self, path=None, page=None): + if not path: + path = self.get_pages_root() + context = {} + request = self.get_request(path, page=page) + context['request'] = request +~~ return Context(context) + + def get_content_renderer(self, request=None): + request = request or self.get_request() + return ContentRenderer(request) + + def get_structure_renderer(self, request=None): + request = request or self.get_request() + return StructureRenderer(request) + + def get_request(self, path=None, language=None, post_data=None, enforce_csrf_checks=False, page=None, domain=None): + factory = RequestFactory() + + if not path: + path = self.get_pages_root() + + if not language: + if settings.USE_I18N: + language = settings.LANGUAGES[0][0] + else: + language = settings.LANGUAGE_CODE + + if post_data: + request = factory.post(path, post_data) + else: + + +## ... source file continues with no further Context examples... + +``` + + +## Example 2 from django-sitetree +[django-sitetree](https://github.com/idlesign/django-sitetree) +([project documentation](https://django-sitetree.readthedocs.io/en/latest/) +and +[PyPI package information](https://pypi.org/project/django-sitetree/)) +is a [Django](/django.html) extension that makes it easier for +developers to add site trees, menus and breadcrumb navigation elements +to their web applications. + +The django-sitetree project is provided as open source under the +[BSD 3-Clause "New" or "Revised" License](https://github.com/idlesign/django-sitetree/blob/master/LICENSE). + +[**django-sitetree / sitetree / sitetreeapp.py**](https://github.com/idlesign/django-sitetree/blob/master/sitetree/./sitetreeapp.py) + +```python +# sitetreeapp.py +import warnings +from collections import defaultdict +from copy import deepcopy +from inspect import getfullargspec +from sys import exc_info +from threading import local +from typing import Callable, List, Optional, Dict, Union, Sequence, Any, Tuple + +from django.conf import settings +from django.core.cache import caches +from django.db.models import signals, QuerySet +from django.template.base import ( + FilterExpression, Lexer, Parser, Variable, VariableDoesNotExist, VARIABLE_TAG_START) +~~from django.template.context import Context +from django.template.loader import get_template +from django.urls import reverse, NoReverseMatch +from django.utils import module_loading +from django.utils.encoding import iri_to_uri +from django.utils.translation import get_language + +from .compat import TOKEN_TEXT, TOKEN_VAR +from .exceptions import SiteTreeError +from .settings import ( + ALIAS_TRUNK, ALIAS_THIS_CHILDREN, ALIAS_THIS_SIBLINGS, ALIAS_THIS_PARENT_SIBLINGS, ALIAS_THIS_ANCESTOR_CHILDREN, + UNRESOLVED_ITEM_MARKER, RAISE_ITEMS_ERRORS_ON_DEBUG, CACHE_TIMEOUT, CACHE_NAME, DYNAMIC_ONLY, ADMIN_APP_NAME, + SITETREE_CLS) +from .utils import get_tree_model, get_tree_item_model, import_app_sitetree_module, generate_id_for + +if False: # pragma: nocover + from django.contrib.auth.models import User # noqa + from .models import TreeItemBase, TreeBase + +TypeDynamicTrees = Dict[str, Union[Dict[str, List['TreeBase']], List['TreeBase']]] + +MODEL_TREE_CLASS = get_tree_model() +MODEL_TREE_ITEM_CLASS = get_tree_item_model() + + + + +## ... source file continues with no further Context examples... + +``` + diff --git a/content/pages/examples/django/django-template-context.markdown b/content/pages/examples/django/django-template-context.markdown new file mode 100644 index 000000000..9bfe9e562 --- /dev/null +++ b/content/pages/examples/django/django-template-context.markdown @@ -0,0 +1,1069 @@ +title: django.template Context Example Code +category: page +slug: django-template-context-examples +sortorder: 500011357 +toc: False +sidebartitle: django.template Context +meta: Example code for understanding how to use the Context class from the django.template module of the Django project. + + +`Context` is a class within the `django.template` module of the Django project. + +Engine, +Library, +Node, +NodeList, +Origin, +RequestContext, +Template, +TemplateDoesNotExist, +TemplateSyntaxError, +Variable, +context, +engine, +library, +and loader +are several other callables with code examples from the same `django.template` package. + +## Example 1 from dccnsys +[dccnsys](https://github.com/dccnconf/dccnsys) is a conference registration +system built with [Django](/django.html). The code is open source under the +[MIT license](https://github.com/dccnconf/dccnsys/blob/master/LICENSE). + +[**dccnsys / wwwdccn / chair_mail / models.py**](https://github.com/dccnconf/dccnsys/blob/master/wwwdccn/chair_mail/models.py) + +```python +# models.py +from django.conf import settings +from django.core.mail import send_mail +from django.db import models +from django.db.models import ForeignKey, OneToOneField, TextField, CharField, \ + SET_NULL, CASCADE, BooleanField, UniqueConstraint +from django.db.models.signals import post_save +from django.dispatch import receiver +~~from django.template import Template, Context +from django.utils import timezone +from markdown import markdown +from html2text import html2text + +from chair_mail.context import get_conference_context, get_user_context, \ + get_submission_context, get_frame_context +from conferences.models import Conference +from submissions.models import Submission +from users.models import User + +MSG_TYPE_USER = 'user' +MSG_TYPE_SUBMISSION = 'submission' + +MESSAGE_TYPE_CHOICES = ( + (MSG_TYPE_USER, 'Message to users'), + (MSG_TYPE_SUBMISSION, 'Message to submissions'), +) + + +class EmailFrame(models.Model): + text_html = models.TextField() + text_plain = models.TextField() + created_at = models.DateTimeField(auto_now_add=True) + updated_at = models.DateTimeField(auto_now_add=True) + created_by = models.ForeignKey(User, on_delete=models.SET_NULL, null=True) + conference = models.ForeignKey(Conference, on_delete=models.CASCADE) + + @staticmethod + def render(frame_template, conference, subject, body): + context_data = get_frame_context(conference, subject, body) +~~ context = Context(context_data, autoescape=False) + return Template(frame_template).render(context) + + def render_html(self, subject, body): + return EmailFrame.render( + self.text_html, self.conference, subject, body + ) + + def render_plain(self, subject, body): + text_plain = self.text_plain + if not text_plain: + text_plain = html2text(self.text_html) + return EmailFrame.render( + text_plain, self.conference, subject, body + ) + + +class EmailSettings(models.Model): + frame = models.ForeignKey(EmailFrame, on_delete=models.SET_NULL, null=True) + conference = models.OneToOneField( + Conference, null=True, blank=True, on_delete=models.CASCADE, + related_name='email_settings', + ) + + + + +## ... source file abbreviated to get to Context examples ... + + + + group_message = models.OneToOneField( + GroupMessage, on_delete=models.CASCADE, parent_link=True) + + @property + def message_type(self): + return MSG_TYPE_USER + + @staticmethod + def create(subject, body, conference, objects_to): + msg = UserMessage.objects.create( + subject=subject, body=body, conference=conference) + for user in objects_to: + msg.recipients.add(user) + msg.save() + return msg + + def send(self, sender): + self.sent = False + self.sent_by = sender + self.save() + + frame = self.conference.email_settings.frame + conference_context = get_conference_context(self.conference) + for user in self.recipients.all(): +~~ context = Context({ + **conference_context, + **get_user_context(user, self.conference) + }, autoescape=False) + email = EmailMessage.create( + group_message=self.group_message, + user_to=user, + context=context, + frame=frame + ) + email.send(sender) + + self.sent_at = timezone.now() + self.sent = True + self.save() + return self + + +class SubmissionMessage(GroupMessage): + recipients = models.ManyToManyField( + Submission, related_name='group_emails') + + group_message = models.OneToOneField( + GroupMessage, on_delete=models.CASCADE, parent_link=True) + + @property + def message_type(self): + return MSG_TYPE_SUBMISSION + + @staticmethod + def create(subject, body, conference, objects_to): + msg = SubmissionMessage.objects.create( + subject=subject, body=body, conference=conference) + for submission in objects_to: + msg.recipients.add(submission) + msg.save() + return msg + + def send(self, sender): + self.sent = False + self.sent_by = sender + self.save() + + frame = self.conference.email_settings.frame + conference_context = get_conference_context(self.conference) + for submission in self.recipients.all(): + submission_context = get_submission_context(submission) + for author in submission.authors.all(): + user = author.user +~~ context = Context({ + **conference_context, + **submission_context, + **get_user_context(user, self.conference) + }, autoescape=False) + email = EmailMessage.create( + group_message=self.group_message, + user_to=user, + context=context, + frame=frame + ) + email.send(sender) + + self.sent_at = timezone.now() + self.sent = True + self.save() + return self + + +def get_group_message_model(msg_type): + return { + MSG_TYPE_USER: UserMessage, + MSG_TYPE_SUBMISSION: SubmissionMessage, + }[msg_type] + + + +## ... source file continues with no further Context examples... + +``` + + +## Example 2 from django-allauth +[django-allauth](https://github.com/pennersr/django-allauth) +([project website](https://www.intenct.nl/projects/django-allauth/)) is a +[Django](/django.html) library for easily adding local and social authentication +flows to Django projects. It is open source under the +[MIT License](https://github.com/pennersr/django-allauth/blob/master/LICENSE). + + +[**django-allauth / allauth / account / tests.py**](https://github.com/pennersr/django-allauth/blob/master/allauth/account/tests.py) + +```python +# tests.py +from __future__ import absolute_import + +import json +import uuid +from datetime import timedelta + +from django import forms +from django.conf import settings +from django.contrib.auth.models import AbstractUser, AnonymousUser +from django.contrib.sites.models import Site +from django.core import mail, validators +from django.core.exceptions import ValidationError +from django.db import models +from django.http import HttpResponseRedirect +~~from django.template import Context, Template +from django.test.client import Client, RequestFactory +from django.test.utils import override_settings +from django.urls import reverse +from django.utils.timezone import now + +from allauth.account.forms import BaseSignupForm, ResetPasswordForm, SignupForm +from allauth.account.models import ( + EmailAddress, + EmailConfirmation, + EmailConfirmationHMAC, +) +from allauth.tests import Mock, TestCase, patch +from allauth.utils import get_user_model, get_username_max_length + +from . import app_settings +from .adapter import get_adapter +from .auth_backends import AuthenticationBackend +from .signals import user_logged_in, user_logged_out +from .utils import ( + filter_users_by_username, + url_str_to_user_pk, + user_pk_to_url_str, + user_username, +) + + +## ... source file continues with no further Context examples... + +``` + + +## Example 3 from django-appmail +[Django-Appmail](https://github.com/yunojuno/django-appmail) +([PyPI package information](https://pypi.org/project/django-appmail/)) +is a [Django](/django.html) app for handling transactional email templates. +While the project began development as a way to work with the Mandrill +transactional [API](/application-programming-interfaces.html), it is +not exclusive to that API. The project simply provides a way to store +and render email content. The library does not send or receive emails. + +Django-Appmail is open sourced under the +[MIT license](https://github.com/yunojuno/django-appmail/blob/master/LICENSE). + +[**django-appmail / appmail / models.py**](https://github.com/yunojuno/django-appmail/blob/master/appmail/./models.py) + +```python +# models.py +from __future__ import annotations + +from typing import Any, Callable, Dict, List, Optional + +from django.conf import settings +from django.contrib.auth import get_user_model +from django.core.exceptions import ValidationError +from django.core.mail import EmailMultiAlternatives +from django.core.serializers.json import DjangoJSONEncoder +from django.db import models, transaction +from django.http import HttpRequest +~~from django.template import Context, Template, TemplateDoesNotExist, TemplateSyntaxError +from django.utils.timezone import now as tz_now +from django.utils.translation import gettext as _ +from django.utils.translation import gettext_lazy as _lazy + +from . import helpers +from .compat import JSONField +from .settings import ( + ADD_EXTRA_HEADERS, + CONTEXT_PROCESSORS, + LOG_SENT_EMAILS, + VALIDATE_ON_SAVE, +) + +User = get_user_model() + + +class EmailTemplateQuerySet(models.query.QuerySet): + def active(self) -> EmailTemplateQuerySet: + return self.filter(is_active=True) + + def current( + self, name: str, language: str = settings.LANGUAGE_CODE + ) -> EmailTemplateQuerySet: + return ( + + +## ... source file abbreviated to get to Context examples ... + + + + def save(self, *args: Any, **kwargs: Any) -> EmailTemplate: + if self.pk is None: + self.test_context = helpers.get_context( + self.subject + self.body_text + self.body_html + ) + validate = kwargs.pop("validate", VALIDATE_ON_SAVE) + if validate: + self.clean() + super(EmailTemplate, self).save(*args, **kwargs) + return self + + def clean(self) -> None: + validation_errors = {} + validation_errors.update(self._validate_body(EmailTemplate.CONTENT_TYPE_PLAIN)) + validation_errors.update(self._validate_body(EmailTemplate.CONTENT_TYPE_HTML)) + validation_errors.update(self._validate_subject()) + if validation_errors: + raise ValidationError(validation_errors) + + def render_subject( + self, + context: dict, + processors: List[Callable[[HttpRequest], dict]] = CONTEXT_PROCESSORS, + ) -> str: +~~ ctx = Context(helpers.patch_context(context, processors), autoescape=False) + return Template(self.subject).render(ctx) + + def _validate_subject(self) -> Dict[str, str]: + try: + self.render_subject({}) + except TemplateDoesNotExist as ex: + return {"subject": _lazy("Template does not exist: {}".format(ex))} + except TemplateSyntaxError as ex: + return {"subject": str(ex)} + else: + return {} + + def render_body( + self, + context: dict, + content_type: str = CONTENT_TYPE_PLAIN, + processors: List[Callable[[HttpRequest], dict]] = CONTEXT_PROCESSORS, + ) -> str: + if content_type not in EmailTemplate.CONTENT_TYPES: + raise ValueError(_(f"Invalid content type. Value supplied: {content_type}")) + if content_type == EmailTemplate.CONTENT_TYPE_PLAIN: +~~ ctx = Context(helpers.patch_context(context, processors), autoescape=False) + return Template(self.body_text).render(ctx) + if content_type == EmailTemplate.CONTENT_TYPE_HTML: +~~ ctx = Context(helpers.patch_context(context, processors)) + return Template(self.body_html).render(ctx) + raise ValueError(f"Invalid content_type '{content_type}'.") + + def _validate_body(self, content_type: str) -> Dict[str, str]: + if content_type == EmailTemplate.CONTENT_TYPE_PLAIN: + field_name = "body_text" + elif content_type == EmailTemplate.CONTENT_TYPE_HTML: + field_name = "body_html" + else: + raise ValueError("Invalid template content_type.") + try: + self.render_body({}, content_type=content_type) + except TemplateDoesNotExist as ex: + return {field_name: _("Template does not exist: {}".format(ex))} + except TemplateSyntaxError as ex: + return {field_name: str(ex)} + else: + return {} + + def clone(self) -> EmailTemplate: + self.pk = None + self.version += 1 + return self.save() + + + +## ... source file continues with no further Context examples... + +``` + + +## Example 4 from django-cms +[django-cms](https://github.com/divio/django-cms) +([project website](https://www.django-cms.org/en/)) is a Python-based +content management system (CMS) [library](https://pypi.org/project/django-cms/) +for use with Django web apps that is open sourced under the +[BSD 3-Clause "New"](https://github.com/divio/django-cms/blob/develop/LICENSE) +license. + +[**django-cms / cms / plugin_rendering.py**](https://github.com/divio/django-cms/blob/develop/cms/./plugin_rendering.py) + +```python +# plugin_rendering.py +from collections import OrderedDict + +from functools import partial + +from classytags.utils import flatten_context + +from django.contrib.sites.models import Site +~~from django.template import Context +from django.utils.functional import cached_property +from django.utils.module_loading import import_string +from django.utils.safestring import mark_safe + +from cms.cache.placeholder import get_placeholder_cache, set_placeholder_cache +from cms.toolbar.utils import ( + get_placeholder_toolbar_js, + get_plugin_toolbar_js, + get_toolbar_from_request, +) +from cms.utils import get_language_from_request +from cms.utils.conf import get_cms_setting +from cms.utils.permissions import has_plugin_permission +from cms.utils.placeholder import get_toolbar_plugin_struct, restore_sekizai_context +from cms.utils.plugins import get_plugin_restrictions + + +def _unpack_plugins(parent_plugin): + found_plugins = [] + + for plugin in parent_plugin.child_plugin_instances or []: + found_plugins.append(plugin) + + if plugin.child_plugin_instances: + + +## ... source file abbreviated to get to Context examples ... + + + placeholder_cache = self._rendered_plugins_by_placeholder.setdefault(instance.placeholder_id, {}) + placeholder_cache.setdefault('plugins', []).append(instance) + return self.get_plugin_toolbar_js(instance, page=page) + + def render_plugins(self, placeholder, language, page=None): + template = page.get_template() if page else None + plugins = self.get_plugins_to_render(placeholder, language, template) + + for plugin in plugins: + plugin._placeholder_cache = placeholder + yield self.render_plugin(plugin, page=page) + + +class LegacyRenderer(ContentRenderer): + + load_structure = True + placeholder_edit_template = ( + ) + + def get_editable_placeholder_context(self, placeholder, page=None): + context = super().get_editable_placeholder_context(placeholder, page) + context['plugin_menu_js'] = self.get_placeholder_plugin_menu(placeholder, page=page) + return context + + +~~class PluginContext(Context): + + def __init__(self, dict_, instance, placeholder, processors=None, current_app=None): + dict_ = flatten_context(dict_) + super().__init__(dict_) + + if not processors: + processors = [] + + for path in get_cms_setting('PLUGIN_CONTEXT_PROCESSORS'): + processor = import_string(path) + self.update(processor(instance, placeholder, self)) + for processor in processors: + self.update(processor(instance, placeholder, self)) + + + +## ... source file continues with no further Context examples... + +``` + + +## Example 5 from django-easy-timezones +[django-easy-timezones](https://github.com/Miserlou/django-easy-timezones) +([project website](https://www.gun.io/blog/django-easy-timezones)) +is a Django +[middleware](https://docs.djangoproject.com/en/2.2/topics/http/middleware/) +[code library](https://pypi.org/project/django-easy-timezones/) +to simplify handling time data in your applications using +users' geolocation data. + +[**django-easy-timezones / easy_timezones / views.py**](https://github.com/Miserlou/django-easy-timezones/blob/master/easy_timezones/./views.py) + +```python +# views.py +from django.conf import settings +from django.http import HttpResponse, HttpResponseRedirect +from django.shortcuts import render_to_response +~~from django.template import RequestContext, Template, Context + +from datetime import datetime + +def with_tz(request): + + dt = datetime.now() + t = Template('{% load tz %}{% localtime on %}{% get_current_timezone as TIME_ZONE %}{{ TIME_ZONE }}{% endlocaltime %}') + c = RequestContext(request) + response = t.render(c) + return HttpResponse(response) + +def without_tz(request): + + t = Template('{% load tz %}{% get_current_timezone as TIME_ZONE %}{{ TIME_ZONE }}') + c = RequestContext(request) + response = t.render(c) + return HttpResponse(response) + + + +## ... source file continues with no further Context examples... + +``` + + +## Example 6 from django-extensions +[django-extensions](https://github.com/django-extensions/django-extensions) +([project documentation](https://django-extensions.readthedocs.io/en/latest/) +and [PyPI page](https://pypi.org/project/django-extensions/)) +is a [Django](/django.html) project that adds a bunch of additional +useful commands to the `manage.py` interface. This +[GoDjango video](https://www.youtube.com/watch?v=1F6G3ONhr4k) provides a +quick overview of what you get when you install it into your Python +environment. + +The django-extensions project is open sourced under the +[MIT license](https://github.com/django-extensions/django-extensions/blob/master/LICENSE). + +[**django-extensions / django_extensions / management / modelviz.py**](https://github.com/django-extensions/django-extensions/blob/master/django_extensions/management/modelviz.py) + +```python +# modelviz.py + +import datetime +import os +import re + +from django.apps import apps +from django.db.models.fields.related import ( + ForeignKey, ManyToManyField, OneToOneField, RelatedField, +) +from django.contrib.contenttypes.fields import GenericRelation +~~from django.template import Context, Template, loader +from django.utils.encoding import force_str +from django.utils.safestring import mark_safe +from django.utils.translation import activate as activate_language + + +__version__ = "1.1" +__license__ = "Python" +__author__ = "Bas van Oostveen ", +__contributors__ = [ + "Antonio Cavedoni " + "Stefano J. Attardi ", + "Carlo C8E Miron", + "Andre Campos ", + "Justin Findlay ", + "Alexander Houben ", + "Joern Hees ", + "Kevin Cherepski ", + "Jose Tomas Tocino ", + "Adam Dobrawy ", + "Mikkel Munch Mortensen ", + "Andrzej Bistram ", + "Daniel Lipsitt ", +] + + +## ... source file abbreviated to get to Context examples ... + + + if '.' in field.remote_field.model: + app_label, model_name = field.remote_field.model.split('.', 1) + else: + app_label = field.model._meta.app_label + model_name = field.remote_field.model + target_model = apps.get_model(app_label, model_name) + else: + target_model = field.remote_field.model + + _rel = self.get_relation_context(target_model, field, label, extras) + + if _rel not in model['relations'] and self.use_model(_rel['target']): + return _rel + + def get_abstract_models(self, appmodels): + abstract_models = [] + for appmodel in appmodels: + abstract_models += [ + abstract_model for abstract_model in appmodel.__bases__ + if hasattr(abstract_model, '_meta') and abstract_model._meta.abstract + ] + abstract_models = list(set(abstract_models)) # remove duplicates + return abstract_models + + def get_app_context(self, app): +~~ return Context({ + 'name': '"%s"' % app.name, + 'app_name': "%s" % app.name, + 'cluster_app_name': "cluster_%s" % app.name.replace(".", "_"), + 'models': [] + }) + + def get_appmodel_attributes(self, appmodel): + if self.relations_as_fields: + attributes = [field for field in appmodel._meta.local_fields] + else: + attributes = [field for field in appmodel._meta.local_fields if not + isinstance(field, RelatedField)] + return attributes + + def get_appmodel_abstracts(self, appmodel): + return [ + abstract_model.__name__ for abstract_model in appmodel.__bases__ + if hasattr(abstract_model, '_meta') and abstract_model._meta.abstract + ] + + def get_appmodel_context(self, appmodel, appmodel_abstracts): + context = { + 'model': appmodel, + 'app_name': appmodel.__module__.replace(".", "_"), + + +## ... source file abbreviated to get to Context examples ... + + + for model_pattern in self.exclude_models: + model_pattern = '^%s$' % model_pattern.replace('*', '.*') + if re.search(model_pattern, model_name): + return False + return not self.include_models + + def skip_field(self, field): + if self.exclude_columns: + if self.verbose_names and field.verbose_name: + if field.verbose_name in self.exclude_columns: + return True + if field.name in self.exclude_columns: + return True + return False + + +def generate_dot(graph_data, template='django_extensions/graph_models/digraph.dot'): + if isinstance(template, str): + template = loader.get_template(template) + + if not isinstance(template, Template) and not (hasattr(template, 'template') and isinstance(template.template, Template)): + raise Exception("Default Django template loader isn't used. " + "This can lead to the incorrect template rendering. " + "Please, check the settings.") + +~~ c = Context(graph_data).flatten() + dot = template.render(c) + + return dot + + +def generate_graph_data(*args, **kwargs): + generator = ModelGraph(*args, **kwargs) + generator.generate_graph_data() + return generator.get_graph_data() + + +def use_model(model, include_models, exclude_models): + generator = ModelGraph([], include_models=include_models, exclude_models=exclude_models) + return generator.use_model(model) + + + +## ... source file continues with no further Context examples... + +``` + + +## Example 7 from django-floppyforms +[django-floppyforms](https://github.com/jazzband/django-floppyforms) +([project documentation](https://django-floppyforms.readthedocs.io/en/latest/) +and +[PyPI page](https://pypi.org/project/django-floppyforms/)) +is a [Django](/django.html) code library for better control +over rendering HTML forms in your [templates](/template-engines.html). + +The django-floppyforms code is provided as +[open source](https://github.com/jazzband/django-floppyforms/blob/master/LICENSE) +and maintained by the collaborative developer community group +[Jazzband](https://jazzband.co/). + +[**django-floppyforms / floppyforms / compat.py**](https://github.com/jazzband/django-floppyforms/blob/master/floppyforms/./compat.py) + +```python +# compat.py +from contextlib import contextmanager + +import django +~~from django.template import Context +from django.utils.datastructures import MultiValueDict + +MULTIVALUE_DICT_TYPES = (MultiValueDict,) + + +REQUIRED_CONTEXT_ATTRIBTUES = ( + '_form_config', + '_form_render', +) + + +class DictContext(dict): + pass + + +if django.VERSION < (1, 8): + def get_template(context, template_name): + from django.template.loader import get_template + return get_template(template_name) + + def get_context(context): +~~ if not isinstance(context, Context): +~~ context = Context(context) + return context + +else: + def get_template(context, template_name): + return context.template.engine.get_template(template_name) + + def get_context(context): + return context + + +def flatten_context(context): +~~ if isinstance(context, Context): + flat = {} + for d in context.dicts: + flat.update(d) + return flat + else: + return context + + +def flatten_contexts(*contexts): + new_context = DictContext() + for context in contexts: + if context is not None: + new_context.update(flatten_context(context)) + for attr in REQUIRED_CONTEXT_ATTRIBTUES: + if hasattr(context, attr): + setattr(new_context, attr, getattr(context, attr)) + return new_context + + +@contextmanager +def render_context(context_instance, context): + if context_instance is not None: + with context_instance.push(context): + yield context_instance + + +## ... source file continues with no further Context examples... + +``` + + +## Example 8 from django-jet +[django-jet](https://github.com/geex-arts/django-jet) +([project documentation](https://jet.readthedocs.io/en/latest/), +[PyPI project page](https://pypi.org/project/django-jet/) and +[more information](http://jet.geex-arts.com/)) +is a fancy [Django](/django.html) Admin panel replacement. + +The django-jet project is open source under the +[GNU Affero General Public License v3.0](https://github.com/geex-arts/django-jet/blob/dev/LICENSE). + +[**django-jet / jet / utils.py**](https://github.com/geex-arts/django-jet/blob/dev/jet/./utils.py) + +```python +# utils.py +import datetime +import json +~~from django.template import Context +from django.utils import translation +from jet import settings +from jet.models import PinnedApplication + +try: + from django.apps.registry import apps +except ImportError: + try: + from django.apps import apps # Fix Django 1.7 import issue + except ImportError: + pass +from django.core.serializers.json import DjangoJSONEncoder +from django.http import HttpResponse +try: + from django.core.urlresolvers import reverse, resolve, NoReverseMatch +except ImportError: # Django 1.11 + from django.urls import reverse, resolve, NoReverseMatch + +from django.contrib.admin import AdminSite +from django.utils.encoding import smart_text +from django.utils.text import capfirst +from django.contrib import messages +from django.utils.encoding import force_text +from django.utils.functional import Promise + + +## ... source file abbreviated to get to Context examples ... + + + item['items'] = item['models'] + return item + app_list = list(map(map_item, original_app_list.values())) + + current_found = False + + for app in app_list: + if not current_found: + for model in app['items']: + if not current_found and model.get('url') and context['request'].path.startswith(model['url']): + model['current'] = True + current_found = True + else: + model['current'] = False + + if not current_found and app.get('url') and context['request'].path.startswith(app['url']): + app['current'] = True + current_found = True + else: + app['current'] = False + + return app_list + + +def context_to_dict(context): +~~ if isinstance(context, Context): + flat = {} + for d in context.dicts: + flat.update(d) + context = flat + + return context + + +def user_is_authenticated(user): + if not hasattr(user.is_authenticated, '__call__'): + return user.is_authenticated + else: + return user.is_authenticated() + + + +## ... source file continues with no further Context examples... + +``` + + +## Example 9 from django-markdown-view +[django-markdown-view](https://github.com/rgs258/django-markdown-view) +([PyPI package information](https://pypi.org/project/django-markdown-view/)) +is a Django extension for serving [Markdown](/markdown.html) files as +[Django templates](/django-templates.html). The project is open +sourced under the +[BSD 3-Clause "New" or "Revised" license](https://github.com/rgs258/django-markdown-view/blob/master/LICENSE). + +[**django-markdown-view / markdown_view / views.py**](https://github.com/rgs258/django-markdown-view/blob/master/markdown_view/./views.py) + +```python +# views.py +import logging + +import markdown +from django.conf import settings +from django.contrib.auth.mixins import LoginRequiredMixin, UserPassesTestMixin +~~from django.template import Engine, Template, Context +from django.utils.safestring import mark_safe +from django.views.generic import TemplateView + +from markdown_view.constants import ( + DEFAULT_MARKDOWN_VIEW_LOADERS, + DEFAULT_MARKDOWN_VIEW_EXTENSIONS, DEFAULT_MARKDOWN_VIEW_TEMPLATE, + DEFAULT_MARKDOWN_VIEW_USE_REQUEST_CONTEXT, DEFAULT_MARKDOWN_VIEW_EXTRA_CONTEXT, + DEFAULT_MARKDOWN_VIEW_TEMPLATE_USE_HIGHLIGHT_JS, DEFAULT_MARKDOWN_VIEW_TEMPLATE_USE_TOC, +) + +logger = logging.getLogger(__name__) + + +class MarkdownView(TemplateView): + file_name = None + + def get_context_data(self, *args, **kwargs): + context = super().get_context_data(*args, **kwargs) + if self.file_name: + engine = Engine(loaders=getattr( + settings, "MARKDOWN_VIEW_LOADERS", DEFAULT_MARKDOWN_VIEW_LOADERS) + ) + template = engine.get_template(self.file_name) + md = markdown.Markdown(extensions=getattr( + settings, + "MARKDOWN_VIEW_EXTENSIONS", + DEFAULT_MARKDOWN_VIEW_EXTENSIONS + )) + template = Template( + "{{% load static %}}{}".format(md.convert(template.source)) + ) + render_context_base = {} + if getattr( + settings, + "MARKDOWN_VIEW_USE_REQUEST_CONTEXT", + DEFAULT_MARKDOWN_VIEW_USE_REQUEST_CONTEXT + ): + render_context_base = context +~~ render_context = Context({ + **render_context_base, + **(getattr( + settings, + "MARKDOWN_VIEW_EXTRA_CONTEXT", + DEFAULT_MARKDOWN_VIEW_EXTRA_CONTEXT + )) + }) + context.update({ + "markdown_content": mark_safe(template.render(render_context)), + "use_highlight_js": getattr( + settings, + "MARKDOWN_VIEW_TEMPLATE_USE_HIGHLIGHT_JS", + DEFAULT_MARKDOWN_VIEW_TEMPLATE_USE_HIGHLIGHT_JS + ), + "use_toc": False, + }) + + if getattr( + settings, + "MARKDOWN_VIEW_TEMPLATE_USE_TOC", + DEFAULT_MARKDOWN_VIEW_TEMPLATE_USE_TOC + ): + context.update({ + "markdown_toc": mark_safe(md.toc), + + +## ... source file continues with no further Context examples... + +``` + + +## Example 10 from django-smithy +[django-smithy](https://github.com/jamiecounsell/django-smithy) is +a [Django](/django.html) code library that allows users to send +HTTP requests from the Django admin user interface. The code for +the project is open source under the +[MIT license](https://github.com/jamiecounsell/django-smithy/blob/master/LICENSE). + +[**django-smithy / smithy / helpers.py**](https://github.com/jamiecounsell/django-smithy/blob/master/smithy/./helpers.py) + +```python +# helpers.py +~~from django.template import Template, Context +from requests_toolbelt.utils import dump + +def render_with_context(template, context): + template = Template(template) +~~ context = Context(context) + return template.render(context) + +def parse_dump_result(fun, obj): + prefixes = dump.PrefixSettings('', '') + try: + result = bytearray() + fun(obj, prefixes, result) + return result.decode('utf-8') + except Exception: + return "Could not parse request as a string" + + + +## ... source file continues with no further Context examples... + +``` + + +## Example 11 from django-tables2 +[django-tables2](https://github.com/jieter/django-tables2) +([projection documentation](https://django-tables2.readthedocs.io/en/latest/) +and +[PyPI page](https://pypi.org/project/django-tables2/)) +is a code library for [Django](/django.html) that simplifies creating and +displaying tables in [Django templates](/django-templates.html), +especially with more advanced features such as pagination and sorting. +The project and its code are +[available as open source](https://github.com/jieter/django-tables2/blob/master/LICENSE). + +[**django-tables2 / django_tables2 / columns / templatecolumn.py**](https://github.com/jieter/django-tables2/blob/master/django_tables2/columns/templatecolumn.py) + +```python +# templatecolumn.py +~~from django.template import Context, Template +from django.template.loader import get_template +from django.utils.html import strip_tags + +from .base import Column, library + + +@library.register +class TemplateColumn(Column): + + empty_values = () + + def __init__(self, template_code=None, template_name=None, extra_context=None, **extra): + super().__init__(**extra) + self.template_code = template_code + self.template_name = template_name + self.extra_context = extra_context or {} + + if not self.template_code and not self.template_name: + raise ValueError("A template must be provided") + + def render(self, record, table, value, bound_column, **kwargs): +~~ context = getattr(table, "context", Context()) + additional_context = { + "default": bound_column.default, + "column": bound_column, + "record": record, + "value": value, + "row_counter": kwargs["bound_row"].row_counter, + } + additional_context.update(self.extra_context) + with context.update(additional_context): + if self.template_code: + return Template(self.template_code).render(context) + else: + return get_template(self.template_name).render(context.flatten()) + + def value(self, **kwargs): + html = super().value(**kwargs) + return strip_tags(html) if isinstance(html, str) else html + + + +## ... source file continues with no further Context examples... + +``` + diff --git a/content/pages/examples/django/django-template-defaultfilters-escape.markdown b/content/pages/examples/django/django-template-defaultfilters-escape.markdown new file mode 100644 index 000000000..b4a4d96b3 --- /dev/null +++ b/content/pages/examples/django/django-template-defaultfilters-escape.markdown @@ -0,0 +1,142 @@ +title: django.template.defaultfilters escape Example Code +category: page +slug: django-template-defaultfilters-escape-examples +sortorder: 500011383 +toc: False +sidebartitle: django.template.defaultfilters escape +meta: Python example code that shows how to use the escape callable from the django.template.defaultfilters module of the Django project. + + +`escape` is a callable within the `django.template.defaultfilters` module of the Django project. + +filesizeformat, +safe, +slugify, +striptags, +title, +and truncatechars +are several other callables with code examples from the same `django.template.defaultfilters` package. + +## Example 1 from django-cms +[django-cms](https://github.com/divio/django-cms) +([project website](https://www.django-cms.org/en/)) is a Python-based +content management system (CMS) [library](https://pypi.org/project/django-cms/) +for use with Django web apps that is open sourced under the +[BSD 3-Clause "New"](https://github.com/divio/django-cms/blob/develop/LICENSE) +license. + +[**django-cms / cms / admin / pageadmin.py**](https://github.com/divio/django-cms/blob/develop/cms/admin/pageadmin.py) + +```python +# pageadmin.py +import uuid + + +import django +from django.contrib.admin.helpers import AdminForm +from django.conf import settings +from django.urls import re_path +from django.contrib import admin, messages +from django.contrib.admin.models import LogEntry, CHANGE +from django.contrib.admin.options import IS_POPUP_VAR +from django.contrib.admin.utils import get_deleted_objects +from django.contrib.contenttypes.models import ContentType +from django.contrib.sites.models import Site +from django.core.exceptions import (ObjectDoesNotExist, + PermissionDenied, ValidationError) +from django.db import router, transaction +from django.db.models import Q, Prefetch +from django.http import ( + HttpResponseRedirect, + HttpResponse, + Http404, + HttpResponseBadRequest, + HttpResponseForbidden, +) +from django.shortcuts import render, get_object_or_404 +~~from django.template.defaultfilters import escape +from django.template.loader import get_template +from django.template.response import SimpleTemplateResponse, TemplateResponse +from django.utils.encoding import force_text +from django.utils.translation import gettext, gettext_lazy as _, get_language +from django.utils.decorators import method_decorator +from django.views.decorators.http import require_POST +from django.http import QueryDict + +from cms import operations +from cms.admin.forms import ( + AddPageForm, + AddPageTypeForm, + AdvancedSettingsForm, + ChangePageForm, + ChangeListForm, + CopyPageForm, + CopyPermissionForm, + DuplicatePageForm, + MovePageForm, + PagePermissionForm, + PublicationDatesForm, +) +from cms.admin.permissionadmin import PERMISSION_ADMIN_INLINES +from cms.admin.placeholderadmin import PlaceholderAdminMixin + + +## ... source file abbreviated to get to escape examples ... + + + site = self.get_site(request) + language = get_site_language_from_request(request, site_id=site.pk) + languages = self._get_site_languages(request, obj) + context.update({ + 'language': language, + 'language_tabs': languages, + 'show_language_tabs': len(list(languages)) > 1 and not context.get('publishing_dates', False), + }) + return context + + def get_preserved_filters(self, request): + preserved_filters_encoded = super().get_preserved_filters(request) + preserved_filters = QueryDict(preserved_filters_encoded).copy() + lang = request.GET.get('language') + + if lang: + preserved_filters.update({ + 'language': lang + }) + + return preserved_filters.urlencode() + + def _get_404_exception(self, object_id): + exception = Http404(_('%(name)s object with primary key %(key)r does not exist.') % { + 'name': force_text(self.opts.verbose_name), +~~ 'key': escape(object_id), + }) + return exception + + def _has_add_permission_from_request(self, request): + site = self.get_site(request) + parent_node_id = request.GET.get('parent_node', None) + + if parent_node_id: + try: + parent_item = self.get_queryset(request).get(node=parent_node_id) + except self.model.DoesNotExist: + return False + else: + parent_item = None + + if parent_item: + has_perm = page_permissions.user_can_add_subpage( + request.user, + target=parent_item, + site=site, + ) + else: + has_perm = page_permissions.user_can_add_page(request.user, site=site) + return has_perm + + +## ... source file continues with no further escape examples... + +``` + diff --git a/content/pages/examples/django/django-template-defaultfilters-filesizeformat.markdown b/content/pages/examples/django/django-template-defaultfilters-filesizeformat.markdown new file mode 100644 index 000000000..6847f62c8 --- /dev/null +++ b/content/pages/examples/django/django-template-defaultfilters-filesizeformat.markdown @@ -0,0 +1,137 @@ +title: django.template.defaultfilters filesizeformat Example Code +category: page +slug: django-template-defaultfilters-filesizeformat-examples +sortorder: 500011384 +toc: False +sidebartitle: django.template.defaultfilters filesizeformat +meta: Python example code that shows how to use the filesizeformat callable from the django.template.defaultfilters module of the Django project. + + +`filesizeformat` is a callable within the `django.template.defaultfilters` module of the Django project. + +escape, +safe, +slugify, +striptags, +title, +and truncatechars +are several other callables with code examples from the same `django.template.defaultfilters` package. + +## Example 1 from wagtail +[wagtail](https://github.com/wagtail/wagtail) +([project website](https://wagtail.io/)) is a fantastic +[Django](/django.html)-based CMS with code that is open source +under the +[BSD 3-Clause "New" or "Revised" License](https://github.com/wagtail/wagtail/blob/master/LICENSE). + +[**wagtail / wagtail / images / fields.py**](https://github.com/wagtail/wagtail/blob/master/wagtail/images/fields.py) + +```python +# fields.py +import os + +import willow + +from django.conf import settings +from django.core.exceptions import ValidationError +from django.forms.fields import ImageField +~~from django.template.defaultfilters import filesizeformat +from django.utils.translation import gettext_lazy as _ + + +ALLOWED_EXTENSIONS = ['gif', 'jpg', 'jpeg', 'png', 'webp'] +SUPPORTED_FORMATS_TEXT = _("GIF, JPEG, PNG, WEBP") + + +class WagtailImageField(ImageField): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + + self.max_upload_size = getattr(settings, 'WAGTAILIMAGES_MAX_UPLOAD_SIZE', 10 * 1024 * 1024) + self.max_image_pixels = getattr(settings, 'WAGTAILIMAGES_MAX_IMAGE_PIXELS', 128 * 1000000) +~~ max_upload_size_text = filesizeformat(self.max_upload_size) + + if self.max_upload_size is not None: + self.help_text = _( + "Supported formats: %(supported_formats)s. Maximum filesize: %(max_upload_size)s." + ) % { + 'supported_formats': SUPPORTED_FORMATS_TEXT, + 'max_upload_size': max_upload_size_text, + } + else: + self.help_text = _( + "Supported formats: %(supported_formats)s." + ) % { + 'supported_formats': SUPPORTED_FORMATS_TEXT, + } + + self.error_messages['invalid_image_extension'] = _( + "Not a supported image format. Supported formats: %s." + ) % SUPPORTED_FORMATS_TEXT + + self.error_messages['invalid_image_known_format'] = _( + "Not a valid %s image." + ) + + self.error_messages['file_too_large'] = _( + + +## ... source file abbreviated to get to filesizeformat examples ... + + + def check_image_file_format(self, f): + extension = os.path.splitext(f.name)[1].lower()[1:] + + if extension not in ALLOWED_EXTENSIONS: + raise ValidationError(self.error_messages['invalid_image_extension'], code='invalid_image_extension') + + image_format = extension.upper() + if image_format == 'JPG': + image_format = 'JPEG' + + internal_image_format = f.image.format.upper() + if internal_image_format == 'MPO': + internal_image_format = 'JPEG' + + if internal_image_format != image_format: + raise ValidationError(self.error_messages['invalid_image_known_format'] % ( + image_format, + ), code='invalid_image_known_format') + + def check_image_file_size(self, f): + if self.max_upload_size is None: + return + + if f.size > self.max_upload_size: + raise ValidationError(self.error_messages['file_too_large'] % ( +~~ filesizeformat(f.size), + ), code='file_too_large') + + def check_image_pixel_size(self, f): + if self.max_image_pixels is None: + return + + image = willow.Image.open(f) + width, height = image.get_size() + frames = image.get_frame_count() + num_pixels = width * height * frames + + if num_pixels > self.max_image_pixels: + raise ValidationError(self.error_messages['file_too_many_pixels'] % ( + num_pixels + ), code='file_too_many_pixels') + + def to_python(self, data): + f = super().to_python(data) + + if f is not None: + self.check_image_file_size(f) + self.check_image_file_format(f) + self.check_image_pixel_size(f) + + + +## ... source file continues with no further filesizeformat examples... + +``` + diff --git a/content/pages/examples/django/django-template-defaultfilters-safe.markdown b/content/pages/examples/django/django-template-defaultfilters-safe.markdown new file mode 100644 index 000000000..b4b6211df --- /dev/null +++ b/content/pages/examples/django/django-template-defaultfilters-safe.markdown @@ -0,0 +1,102 @@ +title: django.template.defaultfilters safe Example Code +category: page +slug: django-template-defaultfilters-safe-examples +sortorder: 500011385 +toc: False +sidebartitle: django.template.defaultfilters safe +meta: Python example code that shows how to use the safe callable from the django.template.defaultfilters module of the Django project. + + +`safe` is a callable within the `django.template.defaultfilters` module of the Django project. + +escape, +filesizeformat, +slugify, +striptags, +title, +and truncatechars +are several other callables with code examples from the same `django.template.defaultfilters` package. + +## Example 1 from django-floppyforms +[django-floppyforms](https://github.com/jazzband/django-floppyforms) +([project documentation](https://django-floppyforms.readthedocs.io/en/latest/) +and +[PyPI page](https://pypi.org/project/django-floppyforms/)) +is a [Django](/django.html) code library for better control +over rendering HTML forms in your [templates](/template-engines.html). + +The django-floppyforms code is provided as +[open source](https://github.com/jazzband/django-floppyforms/blob/master/LICENSE) +and maintained by the collaborative developer community group +[Jazzband](https://jazzband.co/). + +[**django-floppyforms / floppyforms / gis / widgets.py**](https://github.com/jazzband/django-floppyforms/blob/master/floppyforms/gis/widgets.py) + +```python +# widgets.py +from django.conf import settings +~~from django.template.defaultfilters import safe +from django.utils import translation + +import floppyforms as forms + +from urllib.parse import urlencode + +try: + from django.contrib.gis import gdal, geos +except ImportError: + + +__all__ = ('GeometryWidget', 'GeometryCollectionWidget', + 'PointWidget', 'MultiPointWidget', + 'LineStringWidget', 'MultiLineStringWidget', + 'PolygonWidget', 'MultiPolygonWidget', + 'BaseGeometryWidget', 'BaseMetacartaWidget', + 'BaseOsmWidget', 'BaseGMapWidget') + + +class BaseGeometryWidget(forms.Textarea): + display_wkt = False + map_width = 600 + map_height = 400 + map_srid = 4326 + + +## ... source file abbreviated to get to safe examples ... + + + map_srid = 3857 + template_name = 'floppyforms/gis/osm.html' + + class Media: + js = ( + 'floppyforms/openlayers/OpenLayers.js', + 'https://www.openstreetmap.org/openlayers/OpenStreetMap.js', + 'floppyforms/js/MapWidget.js', + ) + + +class BaseGMapWidget(BaseGeometryWidget): + map_srid = 3857 + template_name = 'floppyforms/gis/google.html' + google_maps_api_key = None + + @property + def media(self): + qs_dict = {'v': '3'} + if self.google_maps_api_key is not None: + qs_dict['key'] = self.google_maps_api_key + + js = ( + 'floppyforms/openlayers/OpenLayers.js', + 'floppyforms/js/MapWidget.js', +~~ safe('https://maps.google.com/maps/api/js?' + urlencode(qs_dict)) + ) + return forms.Media(js=js) + + + +## ... source file continues with no further safe examples... + +``` + diff --git a/content/pages/examples/django/django-template-defaultfilters-slugify.markdown b/content/pages/examples/django/django-template-defaultfilters-slugify.markdown new file mode 100644 index 000000000..ab2ba8cc6 --- /dev/null +++ b/content/pages/examples/django/django-template-defaultfilters-slugify.markdown @@ -0,0 +1,276 @@ +title: django.template.defaultfilters slugify Example Code +category: page +slug: django-template-defaultfilters-slugify-examples +sortorder: 500011386 +toc: False +sidebartitle: django.template.defaultfilters slugify +meta: Python example code that shows how to use the slugify callable from the django.template.defaultfilters module of the Django project. + + +`slugify` is a callable within the `django.template.defaultfilters` module of the Django project. + +escape, +filesizeformat, +safe, +striptags, +title, +and truncatechars +are several other callables with code examples from the same `django.template.defaultfilters` package. + +## Example 1 from django-cms +[django-cms](https://github.com/divio/django-cms) +([project website](https://www.django-cms.org/en/)) is a Python-based +content management system (CMS) [library](https://pypi.org/project/django-cms/) +for use with Django web apps that is open sourced under the +[BSD 3-Clause "New"](https://github.com/divio/django-cms/blob/develop/LICENSE) +license. + +[**django-cms / cms / plugin_pool.py**](https://github.com/divio/django-cms/blob/develop/cms/./plugin_pool.py) + +```python +# plugin_pool.py +from operator import attrgetter + +from django.core.exceptions import ImproperlyConfigured +from django.urls import re_path, include +~~from django.template.defaultfilters import slugify +from django.utils.encoding import force_text +from django.utils.functional import cached_property +from django.utils.module_loading import autodiscover_modules +from django.utils.translation import get_language, deactivate_all, activate +from django.template import TemplateDoesNotExist, TemplateSyntaxError + +from cms.exceptions import PluginAlreadyRegistered, PluginNotRegistered +from cms.plugin_base import CMSPluginBase +from cms.utils.conf import get_cms_setting +from cms.utils.helpers import normalize_name + + +class PluginPool: + + def __init__(self): + self.plugins = {} + self.discovered = False + + def _clear_cached(self): + if 'registered_plugins' in self.__dict__: + del self.__dict__['registered_plugins'] + + if 'plugins_with_extra_menu' in self.__dict__: + del self.__dict__['plugins_with_extra_menu'] + + +## ... source file abbreviated to get to slugify examples ... + + + if placeholder: + plugins = (plugin for plugin in plugins + if not plugin.requires_parent_plugin(placeholder, page)) + return sorted(plugins, key=attrgetter('module')) + + def get_text_enabled_plugins(self, placeholder, page): + plugins = set(self.get_all_plugins(placeholder, page)) + plugins.update(self.get_all_plugins(placeholder, page, 'text_only_plugins')) + return sorted((p for p in plugins if p.text_enabled), + key=attrgetter('module', 'name')) + + def get_plugin(self, name): + self.discover_plugins() + return self.plugins[name] + + def get_patterns(self): + self.discover_plugins() + + lang = get_language() + deactivate_all() + + try: + url_patterns = [] + for plugin in self.registered_plugins: + p = plugin() +~~ slug = slugify(force_text(normalize_name(p.__class__.__name__))) + url_patterns += [ + re_path(r'^plugin/%s/' % (slug,), include(p.plugin_urls)), + ] + finally: + activate(lang) + + return url_patterns + + def get_system_plugins(self): + self.discover_plugins() + return [plugin.__name__ for plugin in self.plugins.values() if plugin.system] + + @cached_property + def registered_plugins(self): + return self.get_all_plugins() + + @cached_property + def plugins_with_extra_menu(self): + plugin_classes = [cls for cls in self.registered_plugins + if cls._has_extra_plugin_menu_items] + return plugin_classes + + @cached_property + def plugins_with_extra_placeholder_menu(self): + + +## ... source file continues with no further slugify examples... + +``` + + +## Example 2 from django-filer +[django-filer](https://github.com/divio/django-filer) +([project documentation](https://django-filer.readthedocs.io/en/latest/)) +is a file management library for uploading and organizing files and images +in Django's admin interface. The project's code is available under the +[BSD 3-Clause "New" or "Revised" open source license](https://github.com/divio/django-filer/blob/develop/LICENSE.txt). + +[**django-filer / filer / utils / files.py**](https://github.com/divio/django-filer/blob/develop/filer/utils/files.py) + +```python +# files.py +from __future__ import absolute_import, unicode_literals + +import mimetypes +import os + +from django.http.multipartparser import ( + ChunkIter, SkipFile, StopFutureHandlers, StopUpload, exhaust, +) +~~from django.template.defaultfilters import slugify as slugify_django +from django.utils.encoding import force_text +from django.utils.text import get_valid_filename as get_valid_filename_django + +from unidecode import unidecode + + +class UploadException(Exception): + pass + + +def handle_upload(request): + if not request.method == "POST": + raise UploadException("AJAX request not valid: must be POST") + if request.is_ajax(): + is_raw = True + filename = request.GET.get('qqfile', False) or request.GET.get('filename', False) or '' + + try: + content_length = int(request.META['CONTENT_LENGTH']) + except (IndexError, TypeError, ValueError): + content_length = None + + if content_length < 0: + raise UploadException("Invalid content length: %r" % content_length) + + +## ... source file abbreviated to get to slugify examples ... + + + for i, handler in enumerate(upload_handlers): + file_obj = handler.file_complete(counters[i]) + if file_obj: + upload = file_obj + break + else: + if len(request.FILES) == 1: + upload, filename, is_raw, mime_type = handle_request_files_upload(request) + else: + raise UploadException("AJAX request not valid: Bad Upload") + return upload, filename, is_raw, mime_type + + +def handle_request_files_upload(request): + is_raw = False + upload = list(request.FILES.values())[0] + filename = upload.name + _, iext = os.path.splitext(filename) + mime_type = upload.content_type.lower() + if iext not in mimetypes.guess_all_extensions(mime_type): + msg = "MIME-Type '{mimetype}' does not correspond to file extension of {filename}." + raise UploadException(msg.format(mimetype=mime_type, filename=filename)) + return upload, filename, is_raw, mime_type + + +~~def slugify(string): + return slugify_django(unidecode(force_text(string))) + + +def get_valid_filename(s): + s = get_valid_filename_django(s) + filename, ext = os.path.splitext(s) +~~ filename = slugify(filename) +~~ ext = slugify(ext) + if ext: + return "%s.%s" % (filename, ext) + else: + return "%s" % (filename,) + + + +## ... source file continues with no further slugify examples... + +``` + + +## Example 3 from gadget-board +[gadget-board](https://github.com/mik4el/gadget-board) is a +[Django](/django.html), +[Django REST Framework (DRF)](/django-rest-framework-drf.html) and +[Angular](/angular.html) web application that is open source under the +[Apache2 license](https://github.com/mik4el/gadget-board/blob/master/LICENSE). + +[**gadget-board / web / gadgets / models.py**](https://github.com/mik4el/gadget-board/blob/master/web/gadgets/models.py) + +```python +# models.py +from django.db import models +from django.contrib.postgres.fields import JSONField +~~from django.template.defaultfilters import slugify +from authentication.models import Account + + +class Gadget(models.Model): + name = models.CharField(max_length=40, unique=True) + slug = models.SlugField(null=True, blank=True) + description = models.TextField() + users_can_upload = models.ManyToManyField(Account) + image_name = models.CharField(max_length=140, blank=True) + created_at = models.DateTimeField(auto_now_add=True) + updated_at = models.DateTimeField(auto_now=True) + + @property + def image_url(self): + if self.image_name != "": + return "backend/static/media/{}".format(self.image_name) + else: + return "backend/static/dashboard_icon_big.png" + + def __str__(self): + return self.name + + def save(self, *args, **kwargs): + if not self.id: +~~ self.slug = slugify(self.name) + + super(Gadget, self).save(*args, **kwargs) + + +class GadgetData(models.Model): + gadget = models.ForeignKey(Gadget, db_index=True, on_delete=models.DO_NOTHING) # Add index on filtered fields + data = JSONField() + added_by = models.ForeignKey(Account, on_delete=models.DO_NOTHING) + timestamp = models.DateTimeField(null=True, blank=True, db_index=True) # Add index on filtered fields + created_at = models.DateTimeField(auto_now_add=True) + + def __str__(self): + return '{} {} {}'.format(self.gadget, self.timestamp, self.added_by) + + + +## ... source file continues with no further slugify examples... + +``` + diff --git a/content/pages/examples/django/django-template-defaultfilters-striptags.markdown b/content/pages/examples/django/django-template-defaultfilters-striptags.markdown new file mode 100644 index 000000000..5d837a866 --- /dev/null +++ b/content/pages/examples/django/django-template-defaultfilters-striptags.markdown @@ -0,0 +1,159 @@ +title: django.template.defaultfilters striptags Example Code +category: page +slug: django-template-defaultfilters-striptags-examples +sortorder: 500011387 +toc: False +sidebartitle: django.template.defaultfilters striptags +meta: Python example code that shows how to use the striptags callable from the django.template.defaultfilters module of the Django project. + + +`striptags` is a callable within the `django.template.defaultfilters` module of the Django project. + +escape, +filesizeformat, +safe, +slugify, +title, +and truncatechars +are several other callables with code examples from the same `django.template.defaultfilters` package. + +## Example 1 from django-wiki +[django-wiki](https://github.com/django-wiki/django-wiki) +([project documentation](https://django-wiki.readthedocs.io/en/master/), +[demo](https://demo.django-wiki.org/), +and [PyPI page](https://pypi.org/project/django-wiki/)) +is a wiki system code library for [Django](/django.html) +projects that makes it easier to create user-editable content. +The project aims to provide necessary core features and then +have an easy plugin format for additional features, rather than +having every exhaustive feature built into the core system. +django-wiki is a rewrite of an earlier now-defunct project +named [django-simplewiki](https://code.google.com/p/django-simple-wiki/). + +The code for django-wiki is provided as open source under the +[GNU General Public License 3.0](https://github.com/django-wiki/django-wiki/blob/master/COPYING). + +[**django-wiki / src/wiki / templatetags / wiki_tags.py**](https://github.com/django-wiki/django-wiki/blob/master/src/wiki/templatetags/wiki_tags.py) + +```python +# wiki_tags.py +import re +from urllib.parse import quote as urlquote + +from django import template +from django.apps import apps +from django.conf import settings as django_settings +from django.contrib.contenttypes.models import ContentType +from django.db.models import Model +from django.forms import BaseForm +~~from django.template.defaultfilters import striptags +from django.utils.safestring import mark_safe +from wiki import models +from wiki.conf import settings +from wiki.core.plugins import registry as plugin_registry + +register = template.Library() + + +_cache = {} + + +@register.simple_tag(takes_context=True) +def article_for_object(context, obj): + if not isinstance(obj, Model): + raise TypeError( + "A Wiki article can only be associated to a Django Model " + "instance, not %s" % type(obj) + ) + + content_type = ContentType.objects.get_for_model(obj) + + if True or obj not in _cache: + try: + article = models.ArticleForObject.objects.get( + + +## ... source file abbreviated to get to striptags examples ... + + +@register.inclusion_tag("wiki/includes/form.html", takes_context=True) +def wiki_form(context, form_obj): + if not isinstance(form_obj, BaseForm): + raise TypeError( + "Error including form, it's not a form, it's a %s" % type(form_obj) + ) + context.update({"form": form_obj}) + return context + + +@register.inclusion_tag("wiki/includes/messages.html", takes_context=True) +def wiki_messages(context): + + messages = context.get("messages", []) + for message in messages: + message.css_class = settings.MESSAGE_TAG_CSS_CLASS[message.level] + context.update({"messages": messages}) + return context + + +@register.filter +def get_content_snippet(content, keyword, max_words=30): + + def clean_text(content): + +~~ content = striptags(content) + words = content.split() + + return words + + max_words = int(max_words) + + match_position = content.lower().find(keyword.lower()) + + if match_position != -1: + try: + match_start = content.rindex(" ", 0, match_position) + 1 + except ValueError: + match_start = 0 + try: + match_end = content.index(" ", match_position + len(keyword)) + except ValueError: + match_end = len(content) + all_before = clean_text(content[:match_start]) + match = content[match_start:match_end] + all_after = clean_text(content[match_end:]) + before_words = all_before[-max_words // 2 :] + after_words = all_after[: max_words - len(before_words)] + before = " ".join(before_words) + after = " ".join(after_words) +~~ html = ("%s %s %s" % (before, striptags(match), after)).strip() + kw_p = re.compile(r"(\S*%s\S*)" % keyword, re.IGNORECASE) + html = kw_p.sub(r"\1", html) + + return mark_safe(html) + + return " ".join(clean_text(content)[:max_words]) + + +@register.filter +def can_read(obj, user): + return obj.can_read(user) + + +@register.filter +def can_write(obj, user): + return obj.can_write(user) + + +@register.filter +def can_delete(obj, user): + return obj.can_delete(user) + + +@register.filter + + +## ... source file continues with no further striptags examples... + +``` + diff --git a/content/pages/examples/django/django-template-defaultfilters-title.markdown b/content/pages/examples/django/django-template-defaultfilters-title.markdown new file mode 100644 index 000000000..97ac646e7 --- /dev/null +++ b/content/pages/examples/django/django-template-defaultfilters-title.markdown @@ -0,0 +1,124 @@ +title: django.template.defaultfilters title Example Code +category: page +slug: django-template-defaultfilters-title-examples +sortorder: 500011388 +toc: False +sidebartitle: django.template.defaultfilters title +meta: Python example code that shows how to use the title callable from the django.template.defaultfilters module of the Django project. + + +`title` is a callable within the `django.template.defaultfilters` module of the Django project. + +escape, +filesizeformat, +safe, +slugify, +striptags, +and truncatechars +are several other callables with code examples from the same `django.template.defaultfilters` package. + +## Example 1 from django-cms +[django-cms](https://github.com/divio/django-cms) +([project website](https://www.django-cms.org/en/)) is a Python-based +content management system (CMS) [library](https://pypi.org/project/django-cms/) +for use with Django web apps that is open sourced under the +[BSD 3-Clause "New"](https://github.com/divio/django-cms/blob/develop/LICENSE) +license. + +[**django-cms / cms / models / placeholdermodel.py**](https://github.com/divio/django-cms/blob/develop/cms/models/placeholdermodel.py) + +```python +# placeholdermodel.py + +import warnings + +from datetime import datetime, timedelta + +from django.contrib import admin +from django.db import models +~~from django.template.defaultfilters import title +from django.utils.encoding import force_text +from django.utils.translation import gettext_lazy as _ + +from cms.cache.placeholder import clear_placeholder_cache +from cms.exceptions import LanguageError +from cms.utils import get_site_id +from cms.utils.i18n import get_language_object +from cms.utils.urlutils import admin_reverse +from cms.constants import ( + EXPIRE_NOW, + MAX_EXPIRATION_TTL, + PUBLISHER_STATE_DIRTY, +) +from cms.utils import get_language_from_request +from cms.utils import permissions +from cms.utils.conf import get_cms_setting + + +class Placeholder(models.Model): + slot = models.CharField(_("slot"), max_length=255, db_index=True, editable=False) + default_width = models.PositiveSmallIntegerField(_("width"), null=True, editable=False) + cache_placeholder = True + is_static = False + is_editable = True + + +## ... source file abbreviated to get to title examples ... + + + slot=self.slot, + location=hex(id(self)), + ) + return display + + def clear(self, language=None): + if language: + qs = self.cmsplugin_set.filter(language=language) + else: + qs = self.cmsplugin_set.all() + qs = qs.order_by('-depth').select_related() + for plugin in qs: + inst, cls = plugin.get_plugin_instance() + if inst and getattr(inst, 'cmsplugin_ptr', False): + inst.cmsplugin_ptr._no_reorder = True + inst._no_reorder = True + inst.delete(no_mp=True) + else: + plugin._no_reorder = True + plugin.delete(no_mp=True) + + def get_label(self): + from cms.utils.placeholder import get_placeholder_conf + + template = self.page.get_template() if self.page else None +~~ name = get_placeholder_conf("name", self.slot, template=template, default=title(self.slot)) + name = _(name) + return name + + def get_extra_context(self, template=None): + from cms.utils.placeholder import get_placeholder_conf + return get_placeholder_conf("extra_context", self.slot, template, {}) + + def get_add_url(self): + return self._get_url('add_plugin') + + def get_edit_url(self, plugin_pk): + return self._get_url('edit_plugin', plugin_pk) + + def get_move_url(self): + return self._get_url('move_plugin') + + def get_delete_url(self, plugin_pk): + return self._get_url('delete_plugin', plugin_pk) + + def get_changelist_url(self): + return self._get_url('changelist') + + def get_clear_url(self): + return self._get_url('clear_placeholder', self.pk) + + +## ... source file continues with no further title examples... + +``` + diff --git a/content/pages/examples/django/django-template-defaultfilters-truncatechars.markdown b/content/pages/examples/django/django-template-defaultfilters-truncatechars.markdown new file mode 100644 index 000000000..96cc9ffa6 --- /dev/null +++ b/content/pages/examples/django/django-template-defaultfilters-truncatechars.markdown @@ -0,0 +1,223 @@ +title: django.template.defaultfilters truncatechars Example Code +category: page +slug: django-template-defaultfilters-truncatechars-examples +sortorder: 500011389 +toc: False +sidebartitle: django.template.defaultfilters truncatechars +meta: Python example code that shows how to use the truncatechars callable from the django.template.defaultfilters module of the Django project. + + +`truncatechars` is a callable within the `django.template.defaultfilters` module of the Django project. + +escape, +filesizeformat, +safe, +slugify, +striptags, +and title +are several other callables with code examples from the same `django.template.defaultfilters` package. + +## Example 1 from django-appmail +[Django-Appmail](https://github.com/yunojuno/django-appmail) +([PyPI package information](https://pypi.org/project/django-appmail/)) +is a [Django](/django.html) app for handling transactional email templates. +While the project began development as a way to work with the Mandrill +transactional [API](/application-programming-interfaces.html), it is +not exclusive to that API. The project simply provides a way to store +and render email content. The library does not send or receive emails. + +Django-Appmail is open sourced under the +[MIT license](https://github.com/yunojuno/django-appmail/blob/master/LICENSE). + +[**django-appmail / appmail / admin.py**](https://github.com/yunojuno/django-appmail/blob/master/appmail/./admin.py) + +```python +# admin.py +from __future__ import annotations + +import json +from typing import Optional, Tuple + +from django.contrib import admin, messages +from django.core.exceptions import ValidationError +from django.db.models.query import QuerySet +from django.http import HttpRequest, HttpResponseRedirect +~~from django.template.defaultfilters import truncatechars +from django.urls import reverse +from django.utils.html import format_html +from django.utils.safestring import mark_safe +from django.utils.translation import gettext_lazy as _lazy + +from .compat import JSONField +from .forms import JSONWidget +from .models import EmailTemplate, LoggedMessage + + +class ValidTemplateListFilter(admin.SimpleListFilter): + + title = _lazy("Is valid") + parameter_name = "valid" + + def lookups( + self, request: HttpRequest, model_admin: admin.ModelAdmin + ) -> Tuple[Tuple[str, str], Tuple[str, str]]: + return (("1", _lazy("True")), ("0", _lazy("False"))) + + def queryset(self, request: HttpRequest, queryset: QuerySet) -> QuerySet: + valid_ids = [] + invalid_ids = [] + for obj in queryset: + + +## ... source file abbreviated to get to truncatechars examples ... + + + + exclude = ("html", "context") + + formfield_overrides = {JSONField: {"widget": JSONWidget}} + + list_display = ("to", "template_name", "_subject", "timestamp") + + list_filter = ("timestamp", "template__name", "template__language") + + raw_id_fields = ("user", "template") + + readonly_fields = ( + "to", + "user", + "template", + "template_context", + "subject", + "body", + "render_html", + "timestamp", + ) + + search_fields = ("to", "subject") + + def _subject(self, obj: LoggedMessage) -> str: +~~ return truncatechars(obj.subject, 50) + + def template_name(self, obj: LoggedMessage) -> str: + return obj.template.name + + def template_context(self, obj: LoggedMessage) -> str: + return self.pretty_print(obj.context) + + def render_html(self, obj: LoggedMessage) -> str: + if obj.id is None: + url = "" + else: + url = reverse( + "appmail:render_message_body_html", kwargs={"email_id": obj.id} + ) + return self.iframe(url) + + render_html.short_description = "HTML (rendered)" # type: ignore + render_html.allow_tags = True # type: ignore + + + +## ... source file continues with no further truncatechars examples... + +``` + + +## Example 2 from elasticsearch-django +[elasticsearch-django](https://github.com/yunojuno/elasticsearch-django) +([PyPI package information](https://pypi.org/project/elasticsearch-django/)) +is a [Django](/django.html) app for managing +[ElasticSearch](https://github.com/elastic/elasticsearch) indexes +populated by [Django ORM](/django-orm.html) models. The project is +available as open source under the +[MIT license](https://github.com/yunojuno/elasticsearch-django/blob/master/LICENSE). + +[**elasticsearch-django / elasticsearch_django / admin.py**](https://github.com/yunojuno/elasticsearch-django/blob/master/elasticsearch_django/./admin.py) + +```python +# admin.py +import logging + +import simplejson as json # simplejson supports Decimal serialization +from django.contrib import admin +~~from django.template.defaultfilters import truncatechars, truncatewords +from django.utils.safestring import mark_safe + +from .models import SearchQuery + +logger = logging.getLogger(__name__) + + +def pprint(data: dict) -> str: + pretty = json.dumps(data, sort_keys=True, indent=4, separators=(",", ": ")) + html = pretty.replace(" ", " ").replace("\n", "
") + return mark_safe("%s" % html) + + +class SearchQueryAdmin(admin.ModelAdmin): + + list_display = ( + "id", + "user", + "search_terms_display", + "total_hits_display", + "returned_", + "min_", + "max_", + "reference", + "executed_at", + ) + list_filter = ("index", "query_type") + search_fields = ("search_terms", "user__first_name", "user__last_name", "reference") + exclude = ("hits", "aggregations", "query", "page", "total_hits_") + readonly_fields = ( + "user", + "index", + "search_terms", + "query_type", + "total_hits", + "total_hits_relation", + "returned_", + "min_", + "max_", + "duration", + "query_", + "hits_", + "aggregations_", + "executed_at", + ) + + def search_terms_display(self, instance: SearchQuery) -> str: + raw = instance.search_terms +~~ return truncatechars(truncatewords(raw, 5), 50) + + def query_(self, instance: SearchQuery) -> str: + return pprint(instance.query) + + def max_(self, instance: SearchQuery) -> str: + return "-" if instance.page_size == 0 else str(instance.max_score) + + max_.short_description = "Max score" # type: ignore + + def min_(self, instance: SearchQuery) -> str: + return "-" if instance.page_size == 0 else str(instance.min_score) + + min_.short_description = "Min score" # type: ignore + + def total_hits_display(self, instance: SearchQuery) -> str: + if instance.total_hits_relation == SearchQuery.TotalHitsRelation.ESTIMATE: + return f"{instance.total_hits}*" + return f"{instance.total_hits}" + + def returned_(self, instance: SearchQuery) -> str: + if instance.page_size == 0: + return "-" + return "%i - %i" % (instance.page_from, instance.page_to) + + + +## ... source file continues with no further truncatechars examples... + +``` + diff --git a/content/pages/examples/django/django-template-loader-get-template.markdown b/content/pages/examples/django/django-template-loader-get-template.markdown new file mode 100644 index 000000000..1665b4ca0 --- /dev/null +++ b/content/pages/examples/django/django-template-loader-get-template.markdown @@ -0,0 +1,469 @@ +title: django.template.loader get_template Example Code +category: page +slug: django-template-loader-get-template-examples +sortorder: 500011392 +toc: False +sidebartitle: django.template.loader get_template +meta: Python example code that shows how to use the get_template callable from the django.template.loader module of the Django project. + + +`get_template` is a callable within the `django.template.loader` module of the Django project. + +render_to_string +and +select_template +are a couple of other callables within the `django.template.loader` package that also have code examples. + +## Example 1 from dccnsys +[dccnsys](https://github.com/dccnconf/dccnsys) is a conference registration +system built with [Django](/django.html). The code is open source under the +[MIT license](https://github.com/dccnconf/dccnsys/blob/master/LICENSE). + +[**dccnsys / wwwdccn / chair_mail / views.py**](https://github.com/dccnconf/dccnsys/blob/master/wwwdccn/chair_mail/views.py) + +```python +# views.py +from django.contrib import messages +from django.http import JsonResponse, HttpResponse +from django.shortcuts import render, get_object_or_404, redirect +~~from django.template.loader import get_template +from django.urls import reverse +from django.utils import timezone +from django.views.decorators.http import require_GET, require_POST + +from conferences.utilities import validate_chair_access +from chair_mail.context import USER_VARS, CONFERENCE_VARS, SUBMISSION_VARS, \ + FRAME_VARS +from chair_mail.forms import EmailFrameUpdateForm, EmailFrameTestForm, \ + MessageForm, get_preview_form_class, EditNotificationForm, \ + UpdateNotificationStateForm +from chair_mail.mailing_lists import ALL_LISTS +from chair_mail.models import EmailSettings, EmailFrame, EmailMessage, \ + GroupMessage, MSG_TYPE_USER, MSG_TYPE_SUBMISSION, get_group_message_model, \ + get_message_leaf_model, SystemNotification, DEFAULT_NOTIFICATIONS_DATA +from chair_mail.utility import get_email_frame, get_email_frame_or_404, \ + reverse_preview_url, reverse_list_objects_url, get_object_name, \ + get_object_url +from conferences.models import Conference + + +def _get_grouped_vars(msg_type): + if msg_type == MSG_TYPE_USER: + return ( + ('Conference variables', CONFERENCE_VARS), + + +## ... source file abbreviated to get to get_template examples ... + + + ('Submission variables', SUBMISSION_VARS), + ) + raise ValueError(f'unrecognized message type "{msg_type}"') + + +@require_GET +def overview(request, conf_pk): + conference = get_object_or_404(Conference, pk=conf_pk) + validate_chair_access(request.user, conference) + frame = get_email_frame(conference) + return render(request, 'chair_mail/tab_pages/overview.html', context={ + 'conference': conference, + 'frame': frame, + 'active_tab': 'overview', + }) + + +@require_POST +def create_frame(request, conf_pk): + conference = get_object_or_404(Conference, pk=conf_pk) + validate_chair_access(request.user, conference) + if not hasattr(conference, 'email_settings'): + EmailSettings.objects.create(conference=conference) + email_settings = conference.email_settings + frame = email_settings.frame +~~ template_html = get_template( + 'chair_mail/email/default_frame_html.html').template +~~ template_plain = get_template( + 'chair_mail/email/default_frame_plain.txt').template + if frame: + frame.text_html = template_html.source + frame.text_plain = template_plain.source + frame.created_at = timezone.now() + frame.updated_at = timezone.now() + frame.created_by = request.user + frame.save() + messages.success(request, 'Reset existing frame') + else: + frame = EmailFrame.objects.create( + conference=conference, + created_by=request.user, + text_plain=template_plain.source, + text_html=template_html.source, + ) + email_settings.frame = frame + email_settings.save() + messages.success(request, 'Created new template') + + default_next = reverse('chair_mail:overview', kwargs={'conf_pk': conf_pk}) + next_url = request.GET.get('next', default_next) + return redirect(next_url) + + + +## ... source file continues with no further get_template examples... + +``` + + +## Example 2 from django-cms +[django-cms](https://github.com/divio/django-cms) +([project website](https://www.django-cms.org/en/)) is a Python-based +content management system (CMS) [library](https://pypi.org/project/django-cms/) +for use with Django web apps that is open sourced under the +[BSD 3-Clause "New"](https://github.com/divio/django-cms/blob/develop/LICENSE) +license. + +[**django-cms / cms / templates.py**](https://github.com/divio/django-cms/blob/develop/cms/./templates.py) + +```python +# templates.py +~~from django.template.loader import get_template +from django.utils.functional import cached_property + + +class TemplatesCache: + + def __init__(self): + self._cached_templates = {} + + def get_cached_template(self, template): + if hasattr(template, 'render'): + return template + + if not template in self._cached_templates: +~~ self._cached_templates[template] = get_template(template) + return self._cached_templates[template] + + @cached_property + def drag_item_template(self): +~~ return get_template('cms/toolbar/dragitem.html') + + @cached_property + def placeholder_plugin_menu_template(self): +~~ return get_template('cms/toolbar/dragitem_menu.html') + + @cached_property + def dragbar_template(self): +~~ return get_template('cms/toolbar/dragbar.html') + + + +## ... source file continues with no further get_template examples... + +``` + + +## Example 3 from django-floppyforms +[django-floppyforms](https://github.com/jazzband/django-floppyforms) +([project documentation](https://django-floppyforms.readthedocs.io/en/latest/) +and +[PyPI page](https://pypi.org/project/django-floppyforms/)) +is a [Django](/django.html) code library for better control +over rendering HTML forms in your [templates](/template-engines.html). + +The django-floppyforms code is provided as +[open source](https://github.com/jazzband/django-floppyforms/blob/master/LICENSE) +and maintained by the collaborative developer community group +[Jazzband](https://jazzband.co/). + +[**django-floppyforms / floppyforms / compat.py**](https://github.com/jazzband/django-floppyforms/blob/master/floppyforms/./compat.py) + +```python +# compat.py +from contextlib import contextmanager + +import django +from django.template import Context +from django.utils.datastructures import MultiValueDict + +MULTIVALUE_DICT_TYPES = (MultiValueDict,) + + +REQUIRED_CONTEXT_ATTRIBTUES = ( + '_form_config', + '_form_render', +) + + +class DictContext(dict): + pass + + +if django.VERSION < (1, 8): +~~ def get_template(context, template_name): +~~ from django.template.loader import get_template +~~ return get_template(template_name) + + def get_context(context): + if not isinstance(context, Context): + context = Context(context) + return context + +else: +~~ def get_template(context, template_name): + return context.template.engine.get_template(template_name) + + def get_context(context): + return context + + +def flatten_context(context): + if isinstance(context, Context): + flat = {} + for d in context.dicts: + flat.update(d) + return flat + else: + return context + + +def flatten_contexts(*contexts): + new_context = DictContext() + for context in contexts: + if context is not None: + new_context.update(flatten_context(context)) + for attr in REQUIRED_CONTEXT_ATTRIBTUES: + if hasattr(context, attr): + setattr(new_context, attr, getattr(context, attr)) + + +## ... source file continues with no further get_template examples... + +``` + + +## Example 4 from django-sitetree +[django-sitetree](https://github.com/idlesign/django-sitetree) +([project documentation](https://django-sitetree.readthedocs.io/en/latest/) +and +[PyPI package information](https://pypi.org/project/django-sitetree/)) +is a [Django](/django.html) extension that makes it easier for +developers to add site trees, menus and breadcrumb navigation elements +to their web applications. + +The django-sitetree project is provided as open source under the +[BSD 3-Clause "New" or "Revised" License](https://github.com/idlesign/django-sitetree/blob/master/LICENSE). + +[**django-sitetree / sitetree / sitetreeapp.py**](https://github.com/idlesign/django-sitetree/blob/master/sitetree/./sitetreeapp.py) + +```python +# sitetreeapp.py +import warnings +from collections import defaultdict +from copy import deepcopy +from inspect import getfullargspec +from sys import exc_info +from threading import local +from typing import Callable, List, Optional, Dict, Union, Sequence, Any, Tuple + +from django.conf import settings +from django.core.cache import caches +from django.db.models import signals, QuerySet +from django.template.base import ( + FilterExpression, Lexer, Parser, Variable, VariableDoesNotExist, VARIABLE_TAG_START) +from django.template.context import Context +~~from django.template.loader import get_template +from django.urls import reverse, NoReverseMatch +from django.utils import module_loading +from django.utils.encoding import iri_to_uri +from django.utils.translation import get_language + +from .compat import TOKEN_TEXT, TOKEN_VAR +from .exceptions import SiteTreeError +from .settings import ( + ALIAS_TRUNK, ALIAS_THIS_CHILDREN, ALIAS_THIS_SIBLINGS, ALIAS_THIS_PARENT_SIBLINGS, ALIAS_THIS_ANCESTOR_CHILDREN, + UNRESOLVED_ITEM_MARKER, RAISE_ITEMS_ERRORS_ON_DEBUG, CACHE_TIMEOUT, CACHE_NAME, DYNAMIC_ONLY, ADMIN_APP_NAME, + SITETREE_CLS) +from .utils import get_tree_model, get_tree_item_model, import_app_sitetree_module, generate_id_for + +if False: # pragma: nocover + from django.contrib.auth.models import User # noqa + from .models import TreeItemBase, TreeBase + +TypeDynamicTrees = Dict[str, Union[Dict[str, List['TreeBase']], List['TreeBase']]] + +MODEL_TREE_CLASS = get_tree_model() +MODEL_TREE_ITEM_CLASS = get_tree_item_model() + + +_ITEMS_PROCESSOR: Optional[Callable] = None + + +## ... source file abbreviated to get to get_template examples ... + + + return [] + + tree_items = self.filter_items(self.get_children(tree_alias, None), 'sitetree') + tree_items = self.apply_hook(tree_items, 'sitetree') + self.update_has_children(tree_alias, tree_items, 'sitetree') + + return tree_items + + def children( + self, + parent_item: 'TreeItemBase', + navigation_type: str, + use_template: str, + context: Context + ) -> str: + parent_item = self.resolve_var(parent_item, context) + tree_alias, tree_items = self.get_sitetree(parent_item.tree.alias) + + self.tree_climber(tree_alias, self.get_tree_current_item(tree_alias)) + + tree_items = self.get_children(tree_alias, parent_item) + tree_items = self.filter_items(tree_items, navigation_type) + tree_items = self.apply_hook(tree_items, f'{navigation_type}.children') + self.update_has_children(tree_alias, tree_items, navigation_type) + +~~ my_template = get_template(use_template) + + context.push() + context['sitetree_items'] = tree_items + rendered = my_template.render(context.flatten()) + context.pop() + + return rendered + + def get_children(self, tree_alias: str, item: Optional['TreeItemBase']) -> List['TreeItemBase']: + if not self._current_app_is_admin: + tree_alias = self.resolve_tree_i18n_alias(tree_alias) + + return self.cache.get_entry('parents', tree_alias)[item] + + def update_has_children(self, tree_alias: str, tree_items: List['TreeItemBase'], navigation_type: str): + get_children = self.get_children + filter_items = self.filter_items + apply_hook = self.apply_hook + + for tree_item in tree_items: + children = get_children(tree_alias, tree_item) + children = filter_items(children, navigation_type) + children = apply_hook(children, f'{navigation_type}.has_children') + tree_item.has_children = len(children) > 0 + + +## ... source file continues with no further get_template examples... + +``` + + +## Example 5 from django-tables2 +[django-tables2](https://github.com/jieter/django-tables2) +([projection documentation](https://django-tables2.readthedocs.io/en/latest/) +and +[PyPI page](https://pypi.org/project/django-tables2/)) +is a code library for [Django](/django.html) that simplifies creating and +displaying tables in [Django templates](/django-templates.html), +especially with more advanced features such as pagination and sorting. +The project and its code are +[available as open source](https://github.com/jieter/django-tables2/blob/master/LICENSE). + +[**django-tables2 / django_tables2 / tables.py**](https://github.com/jieter/django-tables2/blob/master/django_tables2/./tables.py) + +```python +# tables.py +import copy +from collections import OrderedDict +from itertools import count + +from django.conf import settings +from django.core.paginator import Paginator +from django.db import models +~~from django.template.loader import get_template +from django.utils.encoding import force_str + +from . import columns +from .config import RequestConfig +from .data import TableData +from .rows import BoundRows +from .utils import Accessor, AttributeDict, OrderBy, OrderByTuple, Sequence + + +class DeclarativeColumnsMetaclass(type): + + def __new__(mcs, name, bases, attrs): + attrs["_meta"] = opts = TableOptions(attrs.get("Meta", None), name) + + cols, remainder = [], {} + for attr_name, attr in attrs.items(): + if isinstance(attr, columns.Column): + attr._explicit = True + cols.append((attr_name, attr)) + else: + remainder[attr_name] = attr + attrs = remainder + + cols.sort(key=lambda x: x[1].creation_counter) + + +## ... source file abbreviated to get to get_template examples ... + + + order_by = self._meta.order_by + if order_by is None: + self._order_by = None + order_by = self.data.ordering + if order_by is not None: + self.order_by = order_by + else: + self.order_by = order_by + self.template_name = template_name + if request: + RequestConfig(request).configure(self) + + self._counter = count() + + def get_top_pinned_data(self): + return None + + def get_bottom_pinned_data(self): + return None + + def before_render(self, request): + return + + def as_html(self, request): + self._counter = count() +~~ template = get_template(self.template_name) + + context = {"table": self, "request": request} + + self.before_render(request) + return template.render(context) + + def as_values(self, exclude_columns=None): + if exclude_columns is None: + exclude_columns = () + + columns = [ + column + for column in self.columns.iterall() + if not (column.column.exclude_from_export or column.name in exclude_columns) + ] + + yield [force_str(column.header, strings_only=True) for column in columns] + + for row in self.rows: + yield [ + force_str(row.get_cell_value(column.name), strings_only=True) for column in columns + ] + + def has_footer(self): + + +## ... source file continues with no further get_template examples... + +``` + diff --git a/content/pages/examples/django/django-template-loader-render-to-string.markdown b/content/pages/examples/django/django-template-loader-render-to-string.markdown new file mode 100644 index 000000000..ac9390c29 --- /dev/null +++ b/content/pages/examples/django/django-template-loader-render-to-string.markdown @@ -0,0 +1,1293 @@ +title: django.template.loader render_to_string Example Code +category: page +slug: django-template-loader-render-to-string-examples +sortorder: 500011393 +toc: False +sidebartitle: django.template.loader render_to_string +meta: Python example code that shows how to use the render_to_string callable from the django.template.loader module of the Django project. + + +`render_to_string` is a callable within the `django.template.loader` module of the Django project. + +get_template +and +select_template +are a couple of other callables within the `django.template.loader` package that also have code examples. + +## Example 1 from dccnsys +[dccnsys](https://github.com/dccnconf/dccnsys) is a conference registration +system built with [Django](/django.html). The code is open source under the +[MIT license](https://github.com/dccnconf/dccnsys/blob/master/LICENSE). + +[**dccnsys / wwwdccn / auth_app / views.py**](https://github.com/dccnconf/dccnsys/blob/master/wwwdccn/auth_app/views.py) + +```python +# views.py +import json +from urllib.request import Request, urlopen +from urllib.parse import urlencode + +from django.conf import settings +from django.contrib.auth import views as auth_views +from django.contrib.auth import get_user_model, login +from django.shortcuts import redirect, render +~~from django.template.loader import render_to_string +from django.core.mail import send_mail + +from .forms import SignUpForm + +User = get_user_model() + + +def signup(request): + if request.method == 'POST': + form = SignUpForm(request.POST) + if form.is_valid(): + recaptcha_response = request.POST.get('g-recaptcha-response') + url = 'https://www.google.com/recaptcha/api/siteverify' + values = { + 'secret': settings.RECAPTCHA_SECRET_KEY, + 'response': recaptcha_response + } + data = urlencode(values).encode() + req = Request(url, data=data) + + response = urlopen(req) + result = json.loads(response.read().decode()) + if result['success']: + user = form.save() + user.is_active = True + user.save() + login(request, user) + + context = { + 'email': user.email, + 'protocol': 'https' if request.is_secure() else "http", + 'domain': request.get_host(), + } +~~ html = render_to_string('auth_app/email/welcome.html', context) +~~ text = render_to_string('auth_app/email/welcome.txt', context) + send_mail( + 'Welcome to DCCN Conference Registration System!', + message=text, + html_message=html, + recipient_list=[user.email], + from_email=settings.DEFAULT_FROM_EMAIL, + fail_silently=False, + ) + return redirect('register') + else: + form = SignUpForm() + return render(request, 'auth_app/signup.html', { + 'site_key': settings.RECAPTCHA_SITE_KEY, + 'form': form, + }) + + +class PasswordResetDoneView(auth_views.PasswordResetDoneView): + template_name = 'auth_app/password_reset_done.html' + + + +## ... source file continues with no further render_to_string examples... + +``` + + +## Example 2 from django-allauth +[django-allauth](https://github.com/pennersr/django-allauth) +([project website](https://www.intenct.nl/projects/django-allauth/)) is a +[Django](/django.html) library for easily adding local and social authentication +flows to Django projects. It is open source under the +[MIT License](https://github.com/pennersr/django-allauth/blob/master/LICENSE). + + +[**django-allauth / allauth / account / adapter.py**](https://github.com/pennersr/django-allauth/blob/master/allauth/account/adapter.py) + +```python +# adapter.py +from __future__ import unicode_literals + +import hashlib +import json +import time +import warnings + +from django import forms +from django.conf import settings +from django.contrib import messages +from django.contrib.auth import ( + authenticate, + get_backends, + login as django_login, + logout as django_logout, +) +from django.contrib.auth.models import AbstractUser +from django.contrib.auth.password_validation import validate_password +from django.contrib.sites.shortcuts import get_current_site +from django.core.cache import cache +from django.core.mail import EmailMessage, EmailMultiAlternatives +from django.http import HttpResponse, HttpResponseRedirect +from django.shortcuts import resolve_url +from django.template import TemplateDoesNotExist +~~from django.template.loader import render_to_string +from django.urls import reverse +from django.utils import timezone +from django.utils.encoding import force_str +from django.utils.translation import gettext_lazy as _ + +from ..utils import ( + build_absolute_uri, + email_address_exists, + generate_unique_username, + get_user_model, + import_attribute, +) +from . import app_settings + + +class DefaultAccountAdapter(object): + + error_messages = { + "username_blacklisted": _( + "Username can not be used. Please use other username." + ), + "username_taken": AbstractUser._meta.get_field("username").error_messages[ + "unique" + ], + + +## ... source file abbreviated to get to render_to_string examples ... + + + def stash_user(self, request, user): + request.session["account_user"] = user + + def unstash_user(self, request): + return request.session.pop("account_user", None) + + def is_email_verified(self, request, email): + ret = False + verified_email = request.session.get("account_verified_email") + if verified_email: + ret = verified_email.lower() == email.lower() + return ret + + def format_email_subject(self, subject): + prefix = app_settings.EMAIL_SUBJECT_PREFIX + if prefix is None: + site = get_current_site(self.request) + prefix = "[{name}] ".format(name=site.name) + return prefix + force_str(subject) + + def get_from_email(self): + return settings.DEFAULT_FROM_EMAIL + + def render_mail(self, template_prefix, email, context): + to = [email] if isinstance(email, str) else email +~~ subject = render_to_string("{0}_subject.txt".format(template_prefix), context) + subject = " ".join(subject.splitlines()).strip() + subject = self.format_email_subject(subject) + + from_email = self.get_from_email() + + bodies = {} + for ext in ["html", "txt"]: + try: + template_name = "{0}_message.{1}".format(template_prefix, ext) +~~ bodies[ext] = render_to_string( + template_name, + context, + self.request, + ).strip() + except TemplateDoesNotExist: + if ext == "txt" and not bodies: + raise + if "txt" in bodies: + msg = EmailMultiAlternatives(subject, bodies["txt"], from_email, to) + if "html" in bodies: + msg.attach_alternative(bodies["html"], "text/html") + else: + msg = EmailMessage(subject, bodies["html"], from_email, to) + msg.content_subtype = "html" # Main content is now text/html + return msg + + def send_mail(self, template_prefix, email, context): + msg = self.render_mail(template_prefix, email, context) + msg.send() + + def get_signup_redirect_url(self, request): + return resolve_url(app_settings.SIGNUP_REDIRECT_URL) + + def get_login_redirect_url(self, request): + + +## ... source file abbreviated to get to render_to_string examples ... + + + min_length = app_settings.PASSWORD_MIN_LENGTH + if min_length and len(password) < min_length: + raise forms.ValidationError( + _("Password must be a minimum of {0} " "characters.").format(min_length) + ) + validate_password(password, user) + return password + + def validate_unique_email(self, email): + if email_address_exists(email): + raise forms.ValidationError(self.error_messages["email_taken"]) + return email + + def add_message( + self, + request, + level, + message_template, + message_context=None, + extra_tags="", + ): + if "django.contrib.messages" in settings.INSTALLED_APPS: + try: + if message_context is None: + message_context = {} +~~ message = render_to_string( + message_template, + message_context, + self.request, + ).strip() + if message: + messages.add_message(request, level, message, extra_tags=extra_tags) + except TemplateDoesNotExist: + pass + + def ajax_response(self, request, response, redirect_to=None, form=None, data=None): + resp = {} + status = response.status_code + + if redirect_to: + status = 200 + resp["location"] = redirect_to + if form: + if request.method == "POST": + if form.is_valid(): + status = 200 + else: + status = 400 + else: + status = 200 + + +## ... source file continues with no further render_to_string examples... + +``` + + +## Example 3 from django-cms +[django-cms](https://github.com/divio/django-cms) +([project website](https://www.django-cms.org/en/)) is a Python-based +content management system (CMS) [library](https://pypi.org/project/django-cms/) +for use with Django web apps that is open sourced under the +[BSD 3-Clause "New"](https://github.com/divio/django-cms/blob/develop/LICENSE) +license. + +[**django-cms / cms / templatetags / cms_tags.py**](https://github.com/divio/django-cms/blob/develop/cms/templatetags/cms_tags.py) + +```python +# cms_tags.py +from collections import namedtuple, OrderedDict +from copy import copy +from datetime import datetime + +from django import template +from django.conf import settings +from django.contrib.sites.models import Site +from django.core.mail import mail_managers +from django.db.models import Model +from django.middleware.common import BrokenLinkEmailsMiddleware +~~from django.template.loader import render_to_string +from django.urls import reverse +from django.utils.encoding import force_text, smart_text +from django.utils.html import escape +from django.utils.http import urlencode +from django.utils.translation import ( + get_language, + override as force_language, + gettext_lazy as _, +) + +from classytags.arguments import (Argument, MultiValueArgument, + MultiKeywordArgument) +from classytags.core import Options, Tag +from classytags.helpers import InclusionTag, AsTag +from classytags.parser import Parser +from classytags.utils import flatten_context +from classytags.values import ListValue, StringValue + +from cms.cache.page import get_page_url_cache, set_page_url_cache +from cms.exceptions import PlaceholderNotFound +from cms.models import Page, Placeholder as PlaceholderModel, CMSPlugin, StaticPlaceholder +from cms.plugin_pool import plugin_pool +from cms.toolbar.utils import get_toolbar_from_request +from cms.utils import get_current_site, get_language_from_request, get_site_id + + +## ... source file abbreviated to get to render_to_string examples ... + + + Argument('edit_fields', default=None, required=False), + Argument('language', default=None, required=False), + Argument('filters', default=None, required=False), + Argument('view_url', default=None, required=False), + Argument('view_method', default=None, required=False), + 'as', + Argument('varname', required=False, resolve=False), + ) + + def __init__(self, parser, tokens): + self.parser = parser + super().__init__(parser, tokens) + + def _is_editable(self, request): + return (request and hasattr(request, 'toolbar') and request.toolbar.edit_mode_active) + + def get_template(self, context, **kwargs): + if self._is_editable(context.get('request', None)): + return self.edit_template + return self.template + + def render_tag(self, context, **kwargs): + context.push() + template = self.get_template(context, **kwargs) + data = self.get_context(context, **kwargs) +~~ output = render_to_string(template, flatten_context(data)).strip() + context.pop() + if kwargs.get('varname'): + context[kwargs['varname']] = output + return '' + else: + return output + + def _get_editable_context(self, context, instance, language, edit_fields, + view_method, view_url, querystring, editmode=True): + request = context['request'] + if hasattr(request, 'toolbar'): + lang = request.toolbar.toolbar_language + else: + lang = get_language() + opts = instance._meta + if getattr(instance, '_deferred', False): + opts = opts.proxy_for_model._meta + with force_language(lang): + extra_context = {} + if edit_fields == 'changelist': + instance.get_plugin_name = u"%s %s list" % (smart_text(_('Edit')), smart_text(opts.verbose_name)) + extra_context['attribute_name'] = 'changelist' + elif editmode: + instance.get_plugin_name = u"%s %s" % (smart_text(_('Edit')), smart_text(opts.verbose_name)) + + +## ... source file abbreviated to get to render_to_string examples ... + + + extra_context = self._get_empty_context(context, instance, None, + language, view_url, + view_method, editmode=False) + extra_context['render_model_add'] = True + return extra_context + + +class CMSEditableObjectAddBlock(CMSEditableObject): + name = 'render_model_add_block' + options = Options( + Argument('instance'), + Argument('language', default=None, required=False), + Argument('view_url', default=None, required=False), + Argument('view_method', default=None, required=False), + 'as', + Argument('varname', required=False, resolve=False), + blocks=[('endrender_model_add_block', 'nodelist')], + ) + + def render_tag(self, context, **kwargs): + context.push() + template = self.get_template(context, **kwargs) + data = self.get_context(context, **kwargs) + data['content'] = kwargs['nodelist'].render(data) + data['rendered_content'] = data['content'] +~~ output = render_to_string(template, flatten_context(data)) + context.pop() + if kwargs.get('varname'): + context[kwargs['varname']] = output + return '' + else: + return output + + def get_context(self, context, **kwargs): + instance = kwargs.pop('instance') + if isinstance(instance, Model) and not instance.pk: + instance.pk = 0 + kwargs.pop('varname') + kwargs.pop('nodelist') + extra_context = self._get_empty_context(context, instance, None, + editmode=False, **kwargs) + extra_context['render_model_add'] = True + return extra_context + + +class CMSEditableObjectBlock(CMSEditableObject): + name = 'render_model_block' + options = Options( + Argument('instance'), + Argument('edit_fields', default=None, required=False), + Argument('language', default=None, required=False), + Argument('view_url', default=None, required=False), + Argument('view_method', default=None, required=False), + 'as', + Argument('varname', required=False, resolve=False), + blocks=[('endrender_model_block', 'nodelist')], + ) + + def render_tag(self, context, **kwargs): + context.push() + template = self.get_template(context, **kwargs) + data = self.get_context(context, **kwargs) + data['content'] = kwargs['nodelist'].render(data) + data['rendered_content'] = data['content'] +~~ output = render_to_string(template, flatten_context(data)) + context.pop() + if kwargs.get('varname'): + context[kwargs['varname']] = output + return '' + else: + return output + + def get_context(self, context, **kwargs): + kwargs.pop('varname') + kwargs.pop('nodelist') + extra_context = self._get_empty_context(context, **kwargs) + extra_context['instance'] = kwargs.get('instance') + extra_context['render_model_block'] = True + return extra_context + + +class StaticPlaceholderNode(Tag): + name = 'static_placeholder' + options = PlaceholderOptions( + Argument('code', required=True), + MultiValueArgument('extra_bits', required=False, resolve=False), + blocks=[ + ('endstatic_placeholder', 'nodelist'), + ] + + +## ... source file continues with no further render_to_string examples... + +``` + + +## Example 4 from django-debug-toolbar +[django-debug-toolbar](https://github.com/jazzband/django-debug-toolbar) +([project documentation](https://github.com/jazzband/django-debug-toolbar) +and [PyPI page](https://pypi.org/project/django-debug-toolbar/)) +grants a developer detailed request-response cycle information while +developing a [Django](/django.html) web application. +The code for django-debug-toolbar is +[open source](https://github.com/jazzband/django-debug-toolbar/blob/master/LICENSE) +and maintained by the developer community group known as +[Jazzband](https://jazzband.co/). + +[**django-debug-toolbar / debug_toolbar / toolbar.py**](https://github.com/jazzband/django-debug-toolbar/blob/master/debug_toolbar/./toolbar.py) + +```python +# toolbar.py + +import uuid +from collections import OrderedDict + +from django.apps import apps +from django.core.exceptions import ImproperlyConfigured +from django.template import TemplateSyntaxError +~~from django.template.loader import render_to_string +from django.urls import path +from django.utils.module_loading import import_string + +from debug_toolbar import settings as dt_settings + + +class DebugToolbar: + def __init__(self, request, get_response): + self.request = request + self.config = dt_settings.get_config().copy() + panels = [] + for panel_class in reversed(self.get_panel_classes()): + panel = panel_class(self, get_response) + panels.append(panel) + if panel.enabled: + get_response = panel.process_request + self.process_request = get_response + self._panels = OrderedDict() + while panels: + panel = panels.pop() + self._panels[panel.panel_id] = panel + self.stats = {} + self.server_timing_stats = {} + self.store_id = None + + + @property + def panels(self): + return list(self._panels.values()) + + @property + def enabled_panels(self): + return [panel for panel in self._panels.values() if panel.enabled] + + def get_panel_by_id(self, panel_id): + return self._panels[panel_id] + + + def render_toolbar(self): + if not self.should_render_panels(): + self.store() + try: + context = {"toolbar": self} +~~ return render_to_string("debug_toolbar/base.html", context) + except TemplateSyntaxError: + if not apps.is_installed("django.contrib.staticfiles"): + raise ImproperlyConfigured( + "The debug toolbar requires the staticfiles contrib app. " + "Add 'django.contrib.staticfiles' to INSTALLED_APPS and " + "define STATIC_URL in your settings." + ) + else: + raise + + def should_render_panels(self): + render_panels = self.config["RENDER_PANELS"] + if render_panels is None: + render_panels = self.request.META["wsgi.multiprocess"] + return render_panels + + + _store = OrderedDict() + + def store(self): + if self.store_id: + return + self.store_id = uuid.uuid4().hex + self._store[self.store_id] = self + + +## ... source file continues with no further render_to_string examples... + +``` + + +## Example 5 from django-filer +[django-filer](https://github.com/divio/django-filer) +([project documentation](https://django-filer.readthedocs.io/en/latest/)) +is a file management library for uploading and organizing files and images +in Django's admin interface. The project's code is available under the +[BSD 3-Clause "New" or "Revised" open source license](https://github.com/divio/django-filer/blob/develop/LICENSE.txt). + +[**django-filer / filer / fields / folder.py**](https://github.com/divio/django-filer/blob/develop/filer/fields/folder.py) + +```python +# folder.py +from __future__ import absolute_import + +import warnings + +from django import forms +from django.contrib.admin.sites import site +from django.contrib.admin.widgets import ForeignKeyRawIdWidget +from django.core.exceptions import ObjectDoesNotExist +from django.db import models +~~from django.template.loader import render_to_string +from django.urls import reverse +from django.utils.http import urlencode +from django.utils.safestring import mark_safe + +from ..models import Folder +from ..utils.compatibility import truncate_words +from ..utils.model_label import get_model_label + + +class AdminFolderWidget(ForeignKeyRawIdWidget): + choices = None + input_type = 'hidden' + is_hidden = False + + def render(self, name, value, attrs=None, renderer=None): + obj = self.obj_for_value(value) + css_id = attrs.get('id') + css_id_folder = "%s_folder" % css_id + css_id_description_txt = "%s_description_txt" % css_id + if attrs is None: + attrs = {} + related_url = None + + if value: + + +## ... source file abbreviated to get to render_to_string examples ... + + + if not related_url: + related_url = reverse('admin:filer-directory_listing-last') + params = self.url_parameters() + params['_pick'] = 'folder' + if params: + url = '?' + urlencode(sorted(params.items())) + else: + url = '' + if 'class' not in attrs: + attrs['class'] = 'vForeignKeyRawIdAdminField' + super_attrs = attrs.copy() + hidden_input = super(ForeignKeyRawIdWidget, self).render(name, value, super_attrs) + + context = { + 'hidden_input': hidden_input, + 'lookup_url': '%s%s' % (related_url, url), + 'lookup_name': name, + 'span_id': css_id_description_txt, + 'object': obj, + 'clear_id': '%s_clear' % css_id, + 'descid': css_id_description_txt, + 'noimg': 'filer/icons/nofile_32x32.png', + 'foldid': css_id_folder, + 'id': css_id, + } +~~ html = render_to_string('admin/filer/widgets/admin_folder.html', context) + return mark_safe(html) + + def label_for_value(self, value): + obj = self.obj_for_value(value) + return ' %s' % truncate_words(obj, 14) + + def obj_for_value(self, value): + if not value: + return None + try: + key = self.rel.get_related_field().name + obj = self.rel.model._default_manager.get(**{key: value}) + except ObjectDoesNotExist: + obj = None + return obj + + class Media(object): + js = ( + 'filer/js/addons/popup_handling.js', + ) + + +class AdminFolderFormField(forms.ModelChoiceField): + widget = AdminFolderWidget + + +## ... source file continues with no further render_to_string examples... + +``` + + +## Example 6 from django-haystack +[django-haystack](https://github.com/django-haystack/django-haystack) +([project website](http://haystacksearch.org/) and +[PyPI page](https://pypi.org/project/django-haystack/)) +is a search abstraction layer that separates the Python search code +in a [Django](/django.html) web application from the search engine +implementation that it runs on, such as +[Apache Solr](http://lucene.apache.org/solr/), +[Elasticsearch](https://www.elastic.co/) +or [Whoosh](https://whoosh.readthedocs.io/en/latest/intro.html). + +The django-haystack project is open source under the +[BSD license](https://github.com/django-haystack/django-haystack/blob/master/LICENSE). + +[**django-haystack / haystack / panels.py**](https://github.com/django-haystack/django-haystack/blob/master/haystack/./panels.py) + +```python +# panels.py +import datetime + +from debug_toolbar.panels import DebugPanel +~~from django.template.loader import render_to_string +from django.utils.translation import ugettext_lazy as _ + +from haystack import connections + + +class HaystackDebugPanel(DebugPanel): + + name = "Haystack" + has_content = True + + def __init__(self, *args, **kwargs): + super(self.__class__, self).__init__(*args, **kwargs) + self._offset = dict( + (alias, len(connections[alias].queries)) + for alias in connections.connections_info.keys() + ) + self._search_time = 0 + self._queries = [] + self._backends = {} + + def nav_title(self): + return _("Haystack") + + def nav_subtitle(self): + + +## ... source file abbreviated to get to render_to_string examples ... + + + if query.get("additional_kwargs"): + if query["additional_kwargs"].get("result_class"): + query["additional_kwargs"]["result_class"] = str( + query["additional_kwargs"]["result_class"] + ) + + try: + query["width_ratio"] = (float(query["time"]) / self._search_time) * 100 + except ZeroDivisionError: + query["width_ratio"] = 0 + + query["start_offset"] = width_ratio_tally + width_ratio_tally += query["width_ratio"] + + context = self.context.copy() + context.update( + { + "backends": sorted( + self._backends.items(), key=lambda x: -x[1]["time_spent"] + ), + "queries": [q for a, q in self._queries], + "sql_time": self._search_time, + } + ) + +~~ return render_to_string("panels/haystack.html", context) + + + +## ... source file continues with no further render_to_string examples... + +``` + + +## Example 7 from django-jet +[django-jet](https://github.com/geex-arts/django-jet) +([project documentation](https://jet.readthedocs.io/en/latest/), +[PyPI project page](https://pypi.org/project/django-jet/) and +[more information](http://jet.geex-arts.com/)) +is a fancy [Django](/django.html) Admin panel replacement. + +The django-jet project is open source under the +[GNU Affero General Public License v3.0](https://github.com/geex-arts/django-jet/blob/dev/LICENSE). + +[**django-jet / jet / dashboard / dashboard.py**](https://github.com/geex-arts/django-jet/blob/dev/jet/dashboard/dashboard.py) + +```python +# dashboard.py +from importlib import import_module +try: + from django.core.urlresolvers import reverse +except ImportError: # Django 1.11 + from django.urls import reverse + +~~from django.template.loader import render_to_string +from jet.dashboard import modules +from jet.dashboard.models import UserDashboardModule +from django.utils.translation import ugettext_lazy as _ +from jet.ordered_set import OrderedSet +from jet.utils import get_admin_site_name, context_to_dict + +try: + from django.template.context_processors import csrf +except ImportError: + from django.core.context_processors import csrf + + +class Dashboard(object): + + columns = 2 + + children = None + + available_children = None + app_label = None + context = None + modules = None + + class Media: + + +## ... source file abbreviated to get to render_to_string examples ... + + + user=self.context['request'].user.pk + ).all() + + if len(module_models) == 0: + module_models = self.create_initial_module_models(self.context['request'].user) + + loaded_modules = [] + + for module_model in module_models: + module_cls = module_model.load_module() + if module_cls is not None: + module = module_cls(model=module_model, context=self.context) + loaded_modules.append(module) + + self.modules = loaded_modules + + def render(self): + context = context_to_dict(self.context) + context.update({ + 'columns': range(self.columns), + 'modules': self.modules, + 'app_label': self.app_label, + }) + context.update(csrf(context['request'])) + +~~ return render_to_string('jet.dashboard/dashboard.html', context) + + def render_tools(self): + context = context_to_dict(self.context) + context.update({ + 'children': self.children, + 'app_label': self.app_label, + 'available_children': self.available_children + }) + context.update(csrf(context['request'])) + +~~ return render_to_string('jet.dashboard/dashboard_tools.html', context) + + def media(self): + unique_css = OrderedSet() + unique_js = OrderedSet() + + for js in getattr(self.Media, 'js', ()): + unique_js.add(js) + for css in getattr(self.Media, 'css', ()): + unique_css.add(css) + + for module in self.modules: + for js in getattr(module.Media, 'js', ()): + unique_js.add(js) + for css in getattr(module.Media, 'css', ()): + unique_css.add(css) + + class Media: + css = list(unique_css) + js = list(unique_js) + + return Media + + +class AppIndexDashboard(Dashboard): + + +## ... source file continues with no further render_to_string examples... + +``` + + +## Example 8 from django-pipeline +[django-pipeline](https://github.com/jazzband/django-pipeline) +([project documentation](https://django-pipeline.readthedocs.io/en/latest/) +and +[PyPI package information](https://pypi.org/project/django-pipeline/)) +is a code library for handling and compressing +[static content assets](/static-content.html) when handling requests in +[Django](/django.html) web applications. + +The django-pipeline project is open sourced under the +[MIT License](https://github.com/jazzband/django-pipeline/blob/master/LICENSE.txt) +and it is maintained by the developer community group +[Jazzband](https://jazzband.co/). + +[**django-pipeline / pipeline / templatetags / pipeline.py**](https://github.com/jazzband/django-pipeline/blob/master/pipeline/templatetags/pipeline.py) + +```python +# pipeline.py +import logging +import subprocess + +from django.contrib.staticfiles.storage import staticfiles_storage + +from django import template +from django.template.base import VariableDoesNotExist +~~from django.template.loader import render_to_string +from django.utils.safestring import mark_safe + +from ..collector import default_collector +from ..conf import settings +from ..exceptions import CompilerError +from ..packager import Packager, PackageNotFound +from ..utils import guess_type + +logger = logging.getLogger(__name__) + +register = template.Library() + + +class PipelineMixin(object): + request = None + _request_var = None + + @property + def request_var(self): + if not self._request_var: + self._request_var = template.Variable('request') + return self._request_var + + def package_for(self, package_name, package_type): + + +## ... source file abbreviated to get to render_to_string examples ... + + + method = getattr(self, f'render_{package_type}') + + return method(package, package.output_filename) + + def render_compressed_sources(self, package, package_name, package_type): + if settings.PIPELINE_COLLECTOR_ENABLED: + default_collector.collect(self.request) + + packager = Packager() + method = getattr(self, f'render_individual_{package_type}') + + try: + paths = packager.compile(package.paths) + except CompilerError as e: + if settings.SHOW_ERRORS_INLINE: + method = getattr(self, f'render_error_{package_type}') + return method(package_name, e) + else: + raise + + templates = packager.pack_templates(package) + + return method(package, paths, templates=templates) + + def render_error(self, package_type, package_name, e): +~~ return render_to_string('pipeline/compile_error.html', { + 'package_type': package_type, + 'package_name': package_name, + 'command': subprocess.list2cmdline(e.command), + 'errors': e.error_output, + }) + + +class StylesheetNode(PipelineMixin, template.Node): + def __init__(self, name): + self.name = name + + def render(self, context): + super(StylesheetNode, self).render(context) + package_name = template.Variable(self.name).resolve(context) + + try: + package = self.package_for(package_name, 'css') + except PackageNotFound: + logger.warn("Package %r is unknown. Check PIPELINE['STYLESHEETS'] in your settings.", package_name) + return '' # fail silently, do not return anything if an invalid group is specified + return self.render_compressed(package, package_name, 'css') + + def render_css(self, package, path): + template_name = package.template_name or "pipeline/css.html" + context = package.extra_context + context.update({ + 'type': guess_type(path, 'text/css'), + 'url': mark_safe(staticfiles_storage.url(path)) + }) +~~ return render_to_string(template_name, context) + + def render_individual_css(self, package, paths, **kwargs): + tags = [self.render_css(package, path) for path in paths] + return '\n'.join(tags) + + def render_error_css(self, package_name, e): + return super(StylesheetNode, self).render_error( + 'CSS', package_name, e) + + +class JavascriptNode(PipelineMixin, template.Node): + def __init__(self, name): + self.name = name + + def render(self, context): + super(JavascriptNode, self).render(context) + package_name = template.Variable(self.name).resolve(context) + + try: + package = self.package_for(package_name, 'js') + except PackageNotFound: + logger.warn("Package %r is unknown. Check PIPELINE['JAVASCRIPT'] in your settings.", package_name) + return '' # fail silently, do not return anything if an invalid group is specified + return self.render_compressed(package, package_name, 'js') + + def render_js(self, package, path): + template_name = package.template_name or "pipeline/js.html" + context = package.extra_context + context.update({ + 'type': guess_type(path, 'text/javascript'), + 'url': mark_safe(staticfiles_storage.url(path)) + }) +~~ return render_to_string(template_name, context) + + def render_inline(self, package, js): + context = package.extra_context + context.update({ + 'source': js + }) +~~ return render_to_string("pipeline/inline_js.html", context) + + def render_individual_js(self, package, paths, templates=None): + tags = [self.render_js(package, js) for js in paths] + if templates: + tags.append(self.render_inline(package, templates)) + return '\n'.join(tags) + + def render_error_js(self, package_name, e): + return super(JavascriptNode, self).render_error( + 'JavaScript', package_name, e) + + +@register.tag +def stylesheet(parser, token): + try: + tag_name, name = token.split_contents() + except ValueError: + raise template.TemplateSyntaxError('%r requires exactly one argument: the name of a group in the PIPELINE.STYLESHEETS setting' % token.split_contents()[0]) + return StylesheetNode(name) + + +@register.tag +def javascript(parser, token): + try: + + +## ... source file continues with no further render_to_string examples... + +``` + + +## Example 9 from django-wiki +[django-wiki](https://github.com/django-wiki/django-wiki) +([project documentation](https://django-wiki.readthedocs.io/en/master/), +[demo](https://demo.django-wiki.org/), +and [PyPI page](https://pypi.org/project/django-wiki/)) +is a wiki system code library for [Django](/django.html) +projects that makes it easier to create user-editable content. +The project aims to provide necessary core features and then +have an easy plugin format for additional features, rather than +having every exhaustive feature built into the core system. +django-wiki is a rewrite of an earlier now-defunct project +named [django-simplewiki](https://code.google.com/p/django-simple-wiki/). + +The code for django-wiki is provided as open source under the +[GNU General Public License 3.0](https://github.com/django-wiki/django-wiki/blob/master/COPYING). + +[**django-wiki / src/wiki / decorators.py**](https://github.com/django-wiki/django-wiki/blob/master/src/wiki/./decorators.py) + +```python +# decorators.py +from functools import wraps +from urllib.parse import quote as urlquote + +from django.http import HttpResponseForbidden +from django.http import HttpResponseNotFound +from django.http import HttpResponseRedirect +from django.shortcuts import get_object_or_404 +from django.shortcuts import redirect +~~from django.template.loader import render_to_string +from django.urls import reverse +from wiki.conf import settings +from wiki.core.exceptions import NoRootURL + + +def response_forbidden(request, article, urlpath, read_denied=False): + if request.user.is_anonymous: + qs = request.META.get("QUERY_STRING", "") + if qs: + qs = urlquote("?" + qs) + else: + qs = "" + return redirect(settings.LOGIN_URL + "?next=" + request.path + qs) + else: + return HttpResponseForbidden( +~~ render_to_string( + "wiki/permission_denied.html", + context={ + "article": article, + "urlpath": urlpath, + "read_denied": read_denied, + }, + request=request, + ) + ) + + +def get_article( # noqa: max-complexity=23 + func=None, + can_read=True, + can_write=False, + deleted_contents=False, + not_locked=False, + can_delete=False, + can_moderate=False, + can_create=False, +): + + def wrapper(request, *args, **kwargs): + from . import models + + path = kwargs.pop("path", None) + article_id = kwargs.pop("article_id", None) + + if path is not None: + try: + urlpath = models.URLPath.get_by_path(path, select_related=True) + except NoRootURL: + return redirect("wiki:root_create") + except models.URLPath.DoesNotExist: + try: + pathlist = list( + filter( + lambda x: x != "", + path.split("/"), + ) + ) + path = "/".join(pathlist[:-1]) + parent = models.URLPath.get_by_path(path) + return HttpResponseRedirect( + reverse("wiki:create", kwargs={"path": parent.path}) + + "?slug=%s" % pathlist[-1].lower() + ) + except models.URLPath.DoesNotExist: + return HttpResponseNotFound( +~~ render_to_string( + "wiki/error.html", + context={"error_type": "ancestors_missing"}, + request=request, + ) + ) + if urlpath.article: + article = urlpath.article + else: + return_url = reverse("wiki:get", kwargs={"path": urlpath.parent.path}) + urlpath.delete() + return HttpResponseRedirect(return_url) + + elif article_id: + articles = models.Article.objects + + article = get_object_or_404(articles, id=article_id) + try: + urlpath = models.URLPath.objects.get(articles__article=article) + except ( + models.URLPath.DoesNotExist, + models.URLPath.MultipleObjectsReturned, + ): + urlpath = None + + + +## ... source file continues with no further render_to_string examples... + +``` + + +## Example 10 from wagtail +[wagtail](https://github.com/wagtail/wagtail) +([project website](https://wagtail.io/)) is a fantastic +[Django](/django.html)-based CMS with code that is open source +under the +[BSD 3-Clause "New" or "Revised" License](https://github.com/wagtail/wagtail/blob/master/LICENSE). + +[**wagtail / wagtail / snippets / widgets.py**](https://github.com/wagtail/wagtail/blob/master/wagtail/snippets/widgets.py) + +```python +# widgets.py +import json + +from django import forms +from django.contrib.admin.utils import quote +~~from django.template.loader import render_to_string +from django.urls import reverse +from django.utils.translation import gettext_lazy as _ + +from wagtail.admin.staticfiles import versioned_static +from wagtail.admin.widgets import AdminChooser +from wagtail.admin.widgets.button import ListingButton + + +class AdminSnippetChooser(AdminChooser): + + def __init__(self, model, **kwargs): + self.target_model = model + name = self.target_model._meta.verbose_name + self.choose_one_text = _('Choose %s') % name + self.choose_another_text = _('Choose another %s') % name + self.link_to_chosen_text = _('Edit this %s') % name + + super().__init__(**kwargs) + + def get_value_data(self, value): + if value is None: + return None + elif isinstance(value, self.target_model): + instance = value + else: # assume instance ID + instance = self.target_model.objects.get(pk=value) + + app_label = self.target_model._meta.app_label + model_name = self.target_model._meta.model_name + quoted_id = quote(instance.pk) + edit_url = reverse('wagtailsnippets:edit', args=[app_label, model_name, quoted_id]) + + return { + 'id': instance.pk, + 'string': str(instance), + 'edit_url': edit_url, + } + + def render_html(self, name, value_data, attrs): + value_data = value_data or {} + + original_field_html = super().render_html(name, value_data.get('id'), attrs) + +~~ return render_to_string("wagtailsnippets/widgets/snippet_chooser.html", { + 'widget': self, + 'original_field_html': original_field_html, + 'attrs': attrs, + 'value': bool(value_data), # only used by chooser.html to identify blank values + 'display_title': value_data.get('string', ''), + 'edit_url': value_data.get('edit_url', ''), + }) + + def render_js_init(self, id_, name, value_data): + model = self.target_model + + return "createSnippetChooser({id}, {model});".format( + id=json.dumps(id_), + model=json.dumps('{app}/{model}'.format( + app=model._meta.app_label, + model=model._meta.model_name))) + + @property + def media(self): + return forms.Media(js=[ + versioned_static('wagtailsnippets/js/snippet-chooser-modal.js'), + versioned_static('wagtailsnippets/js/snippet-chooser.js'), + ]) + + + +## ... source file continues with no further render_to_string examples... + +``` + diff --git a/content/pages/examples/django/django-template-loader-select-template.markdown b/content/pages/examples/django/django-template-loader-select-template.markdown new file mode 100644 index 000000000..1b4a15d0e --- /dev/null +++ b/content/pages/examples/django/django-template-loader-select-template.markdown @@ -0,0 +1,124 @@ +title: django.template.loader select_template Example Code +category: page +slug: django-template-loader-select-template-examples +sortorder: 500011394 +toc: False +sidebartitle: django.template.loader select_template +meta: Python example code that shows how to use the select_template callable from the django.template.loader module of the Django project. + + +`select_template` is a callable within the `django.template.loader` module of the Django project. + +get_template +and +render_to_string +are a couple of other callables within the `django.template.loader` package that also have code examples. + +## Example 1 from django-tables2 +[django-tables2](https://github.com/jieter/django-tables2) +([projection documentation](https://django-tables2.readthedocs.io/en/latest/) +and +[PyPI page](https://pypi.org/project/django-tables2/)) +is a code library for [Django](/django.html) that simplifies creating and +displaying tables in [Django templates](/django-templates.html), +especially with more advanced features such as pagination and sorting. +The project and its code are +[available as open source](https://github.com/jieter/django-tables2/blob/master/LICENSE). + +[**django-tables2 / django_tables2 / templatetags / django_tables2.py**](https://github.com/jieter/django-tables2/blob/master/django_tables2/templatetags/django_tables2.py) + +```python +# django_tables2.py +import re +from collections import OrderedDict + +from django import template +from django.conf import settings +from django.core.exceptions import ImproperlyConfigured +from django.template import Node, TemplateSyntaxError +~~from django.template.loader import get_template, select_template +from django.templatetags.l10n import register as l10n_register +from django.utils.html import escape +from django.utils.http import urlencode + +import django_tables2 as tables +from django_tables2.paginators import LazyPaginator +from django_tables2.utils import AttributeDict + +register = template.Library() +kwarg_re = re.compile(r"(?:(.+)=)?(.+)") +context_processor_error_msg = ( + "Tag {%% %s %%} requires django.template.context_processors.request to be " + "in the template configuration in " + "settings.TEMPLATES[]OPTIONS.context_processors) in order for the included " + "template tags to function correctly." +) + + +def token_kwargs(bits, parser): + if not bits: + return {} + kwargs = OrderedDict() + while bits: + match = kwarg_re.match(bits[0]) + + +## ... source file abbreviated to get to select_template examples ... + + + self.template_name = template_name + + def render(self, context): + table = self.table.resolve(context) + + request = context.get("request") + + if isinstance(table, tables.Table): + pass + elif hasattr(table, "model"): + queryset = table + + table = tables.table_factory(model=queryset.model)(queryset, request=request) + else: + klass = type(table).__name__ + raise ValueError("Expected table or queryset, not {}".format(klass)) + + if self.template_name: + template_name = self.template_name.resolve(context) + else: + template_name = table.template_name + + if isinstance(template_name, str): + template = get_template(template_name) + else: +~~ template = select_template(template_name) + + try: + table.context = context + table.before_render(request) + + return template.render(context={"table": table}, request=request) + finally: + del table.context + + +@register.tag +def render_table(parser, token): + bits = token.split_contents() + bits.pop(0) + + table = parser.compile_filter(bits.pop(0)) + template = parser.compile_filter(bits.pop(0)) if bits else None + + return RenderTableNode(table, template) + + +register.filter("localize", l10n_register.filters["localize"]) +register.filter("unlocalize", l10n_register.filters["unlocalize"]) + + + +## ... source file continues with no further select_template examples... + +``` + diff --git a/content/pages/examples/django/django-template-loader-tags-blocknode.markdown b/content/pages/examples/django/django-template-loader-tags-blocknode.markdown new file mode 100644 index 000000000..908fd9095 --- /dev/null +++ b/content/pages/examples/django/django-template-loader-tags-blocknode.markdown @@ -0,0 +1,192 @@ +title: django.template.loader_tags BlockNode Example Code +category: page +slug: django-template-loader-tags-blocknode-examples +sortorder: 500011395 +toc: False +sidebartitle: django.template.loader_tags BlockNode +meta: Example code for understanding how to use the BlockNode class from the django.template.loader_tags module of the Django project. + + +`BlockNode` is a class within the `django.template.loader_tags` module of the Django project. + +ExtendsNode +and +IncludeNode +are a couple of other callables within the `django.template.loader_tags` package that also have code examples. + +## Example 1 from django-cms +[django-cms](https://github.com/divio/django-cms) +([project website](https://www.django-cms.org/en/)) is a Python-based +content management system (CMS) [library](https://pypi.org/project/django-cms/) +for use with Django web apps that is open sourced under the +[BSD 3-Clause "New"](https://github.com/divio/django-cms/blob/develop/LICENSE) +license. + +[**django-cms / cms / utils / placeholder.py**](https://github.com/divio/django-cms/blob/develop/cms/utils/placeholder.py) + +```python +# placeholder.py +import operator +import warnings +from collections import OrderedDict + +from django.conf import settings +from django.core.exceptions import ImproperlyConfigured +from django.db.models.query_utils import Q +from django.template import TemplateSyntaxError, NodeList, Variable, Context, Template, engines +from django.template.base import VariableNode +from django.template.loader import get_template +~~from django.template.loader_tags import BlockNode, ExtendsNode, IncludeNode + +from sekizai.helpers import get_varname + +from cms.exceptions import DuplicatePlaceholderWarning +from cms.utils.conf import get_cms_setting + + +def _get_nodelist(tpl): + if hasattr(tpl, 'template'): + return tpl.template.nodelist + else: + return tpl.nodelist + + +def get_context(): + if engines is not None: + context = Context() + context.template = Template('') + return context + else: + return {} + + +def get_placeholder_conf(setting, placeholder, template=None, default=None): + + +## ... source file abbreviated to get to BlockNode examples ... + + + validate_placeholder_name(slot) + placeholders.append(placeholder) + clean_placeholders.append(slot) + return placeholders + + +def get_static_placeholders(template, context): + compiled_template = get_template(template) + nodes = _scan_static_placeholders(_get_nodelist(compiled_template)) + placeholders = [node.get_declaration(context) for node in nodes] + placeholders_with_code = [] + + for placeholder in placeholders: + if placeholder.slot: + placeholders_with_code.append(placeholder) + else: + warnings.warn('Unable to resolve static placeholder ' + 'name in template "{}"'.format(template), + Warning) + return placeholders_with_code + + +def _get_block_nodes(extend_node): + parent = extend_node.get_parent(get_context()) + parent_nodelist = _get_nodelist(parent) +~~ parent_nodes = parent_nodelist.get_nodes_by_type(BlockNode) + parent_extend_nodes = parent_nodelist.get_nodes_by_type(ExtendsNode) + + if parent_extend_nodes: + nodes = _get_block_nodes(parent_extend_nodes[0]) + else: + nodes = OrderedDict() + + for node in parent_nodes: + nodes[node.name] = node + +~~ current_nodes = _get_nodelist(extend_node).get_nodes_by_type(BlockNode) + + for node in current_nodes: + if node.name in nodes: + node.super = nodes[node.name] + nodes[node.name] = node + return nodes + + +def _get_placeholder_nodes_from_extend(extend_node, node_class): + block_nodes = _get_block_nodes(extend_node) + block_names = list(block_nodes.keys()) + + placeholders = [] + + for block in block_nodes.values(): + placeholders.extend(_scan_placeholders(_get_nodelist(block), node_class, block, block_names)) + + parent_template = _find_topmost_template(extend_node) + placeholders += _scan_placeholders(_get_nodelist(parent_template), node_class, None, block_names) + return placeholders + + +def _find_topmost_template(extend_node): + parent_template = extend_node.get_parent(get_context()) + nodes.append(node) + elif isinstance(node, IncludeNode): + if node.template: + if not callable(getattr(node.template, 'render', None)): + if isinstance(node.template.var, Variable): + continue + else: + template = get_template(node.template.var) + else: + template = node.template + nodes += _scan_placeholders(_get_nodelist(template), node_class, current_block) + elif isinstance(node, ExtendsNode): + nodes += _get_placeholder_nodes_from_extend(node, node_class) + elif isinstance(node, VariableNode) and current_block: + if node.filter_expression.token == 'block.super': + if not hasattr(current_block.super, 'nodelist'): + raise TemplateSyntaxError("Cannot render block.super for blocks without a parent.") + nodes += _scan_placeholders(_get_nodelist(current_block.super), node_class, current_block.super) + elif isinstance(node, BlockNode) and node.name in ignore_blocks: + continue + elif hasattr(node, 'child_nodelists'): + for nodelist_name in node.child_nodelists: + if hasattr(node, nodelist_name): + subnodelist = getattr(node, nodelist_name) + if isinstance(subnodelist, NodeList): +~~ if isinstance(node, BlockNode): + current_block = node + nodes += _scan_placeholders(subnodelist, node_class, current_block, ignore_blocks) + else: + for attr in dir(node): + obj = getattr(node, attr) + if isinstance(obj, NodeList): +~~ if isinstance(node, BlockNode): + current_block = node + nodes += _scan_placeholders(obj, node_class, current_block, ignore_blocks) + return nodes + + +def _scan_static_placeholders(nodelist): + from cms.templatetags.cms_tags import StaticPlaceholderNode + + return _scan_placeholders(nodelist, node_class=StaticPlaceholderNode) + + +def get_placeholders(template): + compiled_template = get_template(template) + + placeholders = [] + nodes = _scan_placeholders(_get_nodelist(compiled_template)) + clean_placeholders = [] + + for node in nodes: + placeholder = node.get_declaration() + slot = placeholder.slot + + if slot in clean_placeholders: + warnings.warn("Duplicate {{% placeholder \"{0}\" %}} " + + +## ... source file continues with no further BlockNode examples... + +``` + diff --git a/content/pages/examples/django/django-template-loader-tags-extendsnode.markdown b/content/pages/examples/django/django-template-loader-tags-extendsnode.markdown new file mode 100644 index 000000000..61b2dd957 --- /dev/null +++ b/content/pages/examples/django/django-template-loader-tags-extendsnode.markdown @@ -0,0 +1,188 @@ +title: django.template.loader_tags ExtendsNode Example Code +category: page +slug: django-template-loader-tags-extendsnode-examples +sortorder: 500011396 +toc: False +sidebartitle: django.template.loader_tags ExtendsNode +meta: Example code for understanding how to use the ExtendsNode class from the django.template.loader_tags module of the Django project. + + +`ExtendsNode` is a class within the `django.template.loader_tags` module of the Django project. + +BlockNode +and +IncludeNode +are a couple of other callables within the `django.template.loader_tags` package that also have code examples. + +## Example 1 from django-cms +[django-cms](https://github.com/divio/django-cms) +([project website](https://www.django-cms.org/en/)) is a Python-based +content management system (CMS) [library](https://pypi.org/project/django-cms/) +for use with Django web apps that is open sourced under the +[BSD 3-Clause "New"](https://github.com/divio/django-cms/blob/develop/LICENSE) +license. + +[**django-cms / cms / utils / placeholder.py**](https://github.com/divio/django-cms/blob/develop/cms/utils/placeholder.py) + +```python +# placeholder.py +import operator +import warnings +from collections import OrderedDict + +from django.conf import settings +from django.core.exceptions import ImproperlyConfigured +from django.db.models.query_utils import Q +from django.template import TemplateSyntaxError, NodeList, Variable, Context, Template, engines +from django.template.base import VariableNode +from django.template.loader import get_template +~~from django.template.loader_tags import BlockNode, ExtendsNode, IncludeNode + +from sekizai.helpers import get_varname + +from cms.exceptions import DuplicatePlaceholderWarning +from cms.utils.conf import get_cms_setting + + +def _get_nodelist(tpl): + if hasattr(tpl, 'template'): + return tpl.template.nodelist + else: + return tpl.nodelist + + +def get_context(): + if engines is not None: + context = Context() + context.template = Template('') + return context + else: + return {} + + +def get_placeholder_conf(setting, placeholder, template=None, default=None): + + +## ... source file abbreviated to get to ExtendsNode examples ... + + + placeholders.append(placeholder) + clean_placeholders.append(slot) + return placeholders + + +def get_static_placeholders(template, context): + compiled_template = get_template(template) + nodes = _scan_static_placeholders(_get_nodelist(compiled_template)) + placeholders = [node.get_declaration(context) for node in nodes] + placeholders_with_code = [] + + for placeholder in placeholders: + if placeholder.slot: + placeholders_with_code.append(placeholder) + else: + warnings.warn('Unable to resolve static placeholder ' + 'name in template "{}"'.format(template), + Warning) + return placeholders_with_code + + +def _get_block_nodes(extend_node): + parent = extend_node.get_parent(get_context()) + parent_nodelist = _get_nodelist(parent) + parent_nodes = parent_nodelist.get_nodes_by_type(BlockNode) +~~ parent_extend_nodes = parent_nodelist.get_nodes_by_type(ExtendsNode) + + if parent_extend_nodes: + nodes = _get_block_nodes(parent_extend_nodes[0]) + else: + nodes = OrderedDict() + + for node in parent_nodes: + nodes[node.name] = node + + current_nodes = _get_nodelist(extend_node).get_nodes_by_type(BlockNode) + + for node in current_nodes: + if node.name in nodes: + node.super = nodes[node.name] + nodes[node.name] = node + return nodes + + +def _get_placeholder_nodes_from_extend(extend_node, node_class): + block_nodes = _get_block_nodes(extend_node) + block_names = list(block_nodes.keys()) + + placeholders = [] + + for block in block_nodes.values(): + placeholders.extend(_scan_placeholders(_get_nodelist(block), node_class, block, block_names)) + + parent_template = _find_topmost_template(extend_node) + placeholders += _scan_placeholders(_get_nodelist(parent_template), node_class, None, block_names) + return placeholders + + +def _find_topmost_template(extend_node): + parent_template = extend_node.get_parent(get_context()) +~~ for node in _get_nodelist(parent_template).get_nodes_by_type(ExtendsNode): + return _find_topmost_template(node) + return extend_node.get_parent(get_context()) + + +def _scan_placeholders(nodelist, node_class=None, current_block=None, ignore_blocks=None): + from cms.templatetags.cms_tags import Placeholder + + if not node_class: + node_class = Placeholder + + nodes = [] + + if ignore_blocks is None: + ignore_blocks = [] + + for node in nodelist: + if isinstance(node, node_class): + nodes.append(node) + elif isinstance(node, IncludeNode): + if node.template: + if not callable(getattr(node.template, 'render', None)): + if isinstance(node.template.var, Variable): + continue + else: + template = get_template(node.template.var) + else: + template = node.template + nodes += _scan_placeholders(_get_nodelist(template), node_class, current_block) +~~ elif isinstance(node, ExtendsNode): + nodes += _get_placeholder_nodes_from_extend(node, node_class) + elif isinstance(node, VariableNode) and current_block: + if node.filter_expression.token == 'block.super': + if not hasattr(current_block.super, 'nodelist'): + raise TemplateSyntaxError("Cannot render block.super for blocks without a parent.") + nodes += _scan_placeholders(_get_nodelist(current_block.super), node_class, current_block.super) + elif isinstance(node, BlockNode) and node.name in ignore_blocks: + continue + elif hasattr(node, 'child_nodelists'): + for nodelist_name in node.child_nodelists: + if hasattr(node, nodelist_name): + subnodelist = getattr(node, nodelist_name) + if isinstance(subnodelist, NodeList): + if isinstance(node, BlockNode): + current_block = node + nodes += _scan_placeholders(subnodelist, node_class, current_block, ignore_blocks) + else: + for attr in dir(node): + obj = getattr(node, attr) + if isinstance(obj, NodeList): + if isinstance(node, BlockNode): + current_block = node + nodes += _scan_placeholders(obj, node_class, current_block, ignore_blocks) + return nodes + + +## ... source file continues with no further ExtendsNode examples... + +``` + diff --git a/content/pages/examples/django/django-template-loader-tags-includenode.markdown b/content/pages/examples/django/django-template-loader-tags-includenode.markdown new file mode 100644 index 000000000..412c41d09 --- /dev/null +++ b/content/pages/examples/django/django-template-loader-tags-includenode.markdown @@ -0,0 +1,124 @@ +title: django.template.loader_tags IncludeNode Example Code +category: page +slug: django-template-loader-tags-includenode-examples +sortorder: 500011397 +toc: False +sidebartitle: django.template.loader_tags IncludeNode +meta: Example code for understanding how to use the IncludeNode class from the django.template.loader_tags module of the Django project. + + +`IncludeNode` is a class within the `django.template.loader_tags` module of the Django project. + +BlockNode +and +ExtendsNode +are a couple of other callables within the `django.template.loader_tags` package that also have code examples. + +## Example 1 from django-cms +[django-cms](https://github.com/divio/django-cms) +([project website](https://www.django-cms.org/en/)) is a Python-based +content management system (CMS) [library](https://pypi.org/project/django-cms/) +for use with Django web apps that is open sourced under the +[BSD 3-Clause "New"](https://github.com/divio/django-cms/blob/develop/LICENSE) +license. + +[**django-cms / cms / utils / placeholder.py**](https://github.com/divio/django-cms/blob/develop/cms/utils/placeholder.py) + +```python +# placeholder.py +import operator +import warnings +from collections import OrderedDict + +from django.conf import settings +from django.core.exceptions import ImproperlyConfigured +from django.db.models.query_utils import Q +from django.template import TemplateSyntaxError, NodeList, Variable, Context, Template, engines +from django.template.base import VariableNode +from django.template.loader import get_template +~~from django.template.loader_tags import BlockNode, ExtendsNode, IncludeNode + +from sekizai.helpers import get_varname + +from cms.exceptions import DuplicatePlaceholderWarning +from cms.utils.conf import get_cms_setting + + +def _get_nodelist(tpl): + if hasattr(tpl, 'template'): + return tpl.template.nodelist + else: + return tpl.nodelist + + +def get_context(): + if engines is not None: + context = Context() + context.template = Template('') + return context + else: + return {} + + +def get_placeholder_conf(setting, placeholder, template=None, default=None): + + +## ... source file abbreviated to get to IncludeNode examples ... + + + + +def restore_sekizai_context(context, changes): + varname = get_varname() + sekizai_container = context.get(varname) + for key, values in changes.items(): + sekizai_namespace = sekizai_container[key] + for value in values: + sekizai_namespace.append(value) + + +def _scan_placeholders(nodelist, node_class=None, current_block=None, ignore_blocks=None): + from cms.templatetags.cms_tags import Placeholder + + if not node_class: + node_class = Placeholder + + nodes = [] + + if ignore_blocks is None: + ignore_blocks = [] + + for node in nodelist: + if isinstance(node, node_class): + nodes.append(node) +~~ elif isinstance(node, IncludeNode): + if node.template: + if not callable(getattr(node.template, 'render', None)): + if isinstance(node.template.var, Variable): + continue + else: + template = get_template(node.template.var) + else: + template = node.template + nodes += _scan_placeholders(_get_nodelist(template), node_class, current_block) + elif isinstance(node, ExtendsNode): + nodes += _get_placeholder_nodes_from_extend(node, node_class) + elif isinstance(node, VariableNode) and current_block: + if node.filter_expression.token == 'block.super': + if not hasattr(current_block.super, 'nodelist'): + raise TemplateSyntaxError("Cannot render block.super for blocks without a parent.") + nodes += _scan_placeholders(_get_nodelist(current_block.super), node_class, current_block.super) + elif isinstance(node, BlockNode) and node.name in ignore_blocks: + continue + elif hasattr(node, 'child_nodelists'): + for nodelist_name in node.child_nodelists: + if hasattr(node, nodelist_name): + subnodelist = getattr(node, nodelist_name) + if isinstance(subnodelist, NodeList): + if isinstance(node, BlockNode): + + +## ... source file continues with no further IncludeNode examples... + +``` + diff --git a/content/pages/examples/django/django-template-loader.markdown b/content/pages/examples/django/django-template-loader.markdown new file mode 100644 index 000000000..9098f4881 --- /dev/null +++ b/content/pages/examples/django/django-template-loader.markdown @@ -0,0 +1,1118 @@ +title: django.template loader Example Code +category: page +slug: django-template-loader-examples +sortorder: 500011368 +toc: False +sidebartitle: django.template loader +meta: Python example code that shows how to use the loader callable from the django.template module of the Django project. + + +`loader` is a callable within the `django.template` module of the Django project. + +Context, +Engine, +Library, +Node, +NodeList, +Origin, +RequestContext, +Template, +TemplateDoesNotExist, +TemplateSyntaxError, +Variable, +context, +engine, +and library +are several other callables with code examples from the same `django.template` package. + +## Example 1 from django-cms +[django-cms](https://github.com/divio/django-cms) +([project website](https://www.django-cms.org/en/)) is a Python-based +content management system (CMS) [library](https://pypi.org/project/django-cms/) +for use with Django web apps that is open sourced under the +[BSD 3-Clause "New"](https://github.com/divio/django-cms/blob/develop/LICENSE) +license. + +[**django-cms / cms / plugin_pool.py**](https://github.com/divio/django-cms/blob/develop/cms/./plugin_pool.py) + +```python +# plugin_pool.py +from operator import attrgetter + +from django.core.exceptions import ImproperlyConfigured +from django.urls import re_path, include +from django.template.defaultfilters import slugify +from django.utils.encoding import force_text +from django.utils.functional import cached_property +from django.utils.module_loading import autodiscover_modules +from django.utils.translation import get_language, deactivate_all, activate +from django.template import TemplateDoesNotExist, TemplateSyntaxError + +from cms.exceptions import PluginAlreadyRegistered, PluginNotRegistered +from cms.plugin_base import CMSPluginBase +from cms.utils.conf import get_cms_setting +from cms.utils.helpers import normalize_name + + +class PluginPool: + + def __init__(self): + self.plugins = {} + self.discovered = False + + def _clear_cached(self): + if 'registered_plugins' in self.__dict__: + del self.__dict__['registered_plugins'] + + if 'plugins_with_extra_menu' in self.__dict__: + del self.__dict__['plugins_with_extra_menu'] + + if 'plugins_with_extra_placeholder_menu' in self.__dict__: + del self.__dict__['plugins_with_extra_placeholder_menu'] + + def discover_plugins(self): + if self.discovered: + + +## ... source file abbreviated to get to loader examples ... + + + autodiscover_modules('cms_plugins') + self.discovered = True + + def clear(self): + self.discovered = False + self.plugins = {} + self._clear_cached() + + def validate_templates(self, plugin=None): + if plugin: + plugins = [plugin] + else: + plugins = self.plugins.values() + for plugin in plugins: + if (plugin.render_plugin and not type(plugin.render_plugin) == property + or hasattr(plugin.model, 'render_template') + or hasattr(plugin, 'get_render_template')): + if (plugin.render_template is None and + not hasattr(plugin, 'get_render_template')): + raise ImproperlyConfigured( + "CMS Plugins must define a render template, " + "a get_render_template method or " + "set render_plugin=False: %s" % plugin + ) + elif not hasattr(plugin, 'get_render_template'): +~~ from django.template import loader + + template = plugin.render_template + if isinstance(template, str) and template: + try: +~~ loader.get_template(template) + except TemplateDoesNotExist as e: + if str(e) == template: + raise ImproperlyConfigured( + "CMS Plugins must define a render template (%s) that exists: %s" + % (plugin, template) + ) + else: + pass + except TemplateSyntaxError: + pass + else: + if plugin.allow_children: + raise ImproperlyConfigured( + "CMS Plugins can not define render_plugin=False and allow_children=True: %s" + % plugin + ) + + def register_plugin(self, plugin): + if not issubclass(plugin, CMSPluginBase): + raise ImproperlyConfigured( + "CMS Plugins must be subclasses of CMSPluginBase, %r is not." + % plugin + ) + plugin_name = plugin.__name__ + + +## ... source file continues with no further loader examples... + +``` + + +## Example 2 from django-extensions +[django-extensions](https://github.com/django-extensions/django-extensions) +([project documentation](https://django-extensions.readthedocs.io/en/latest/) +and [PyPI page](https://pypi.org/project/django-extensions/)) +is a [Django](/django.html) project that adds a bunch of additional +useful commands to the `manage.py` interface. This +[GoDjango video](https://www.youtube.com/watch?v=1F6G3ONhr4k) provides a +quick overview of what you get when you install it into your Python +environment. + +The django-extensions project is open sourced under the +[MIT license](https://github.com/django-extensions/django-extensions/blob/master/LICENSE). + +[**django-extensions / django_extensions / management / modelviz.py**](https://github.com/django-extensions/django-extensions/blob/master/django_extensions/management/modelviz.py) + +```python +# modelviz.py + +import datetime +import os +import re + +from django.apps import apps +from django.db.models.fields.related import ( + ForeignKey, ManyToManyField, OneToOneField, RelatedField, +) +from django.contrib.contenttypes.fields import GenericRelation +~~from django.template import Context, Template, loader +from django.utils.encoding import force_str +from django.utils.safestring import mark_safe +from django.utils.translation import activate as activate_language + + +__version__ = "1.1" +__license__ = "Python" +__author__ = "Bas van Oostveen ", +__contributors__ = [ + "Antonio Cavedoni " + "Stefano J. Attardi ", + "Carlo C8E Miron", + "Andre Campos ", + "Justin Findlay ", + "Alexander Houben ", + "Joern Hees ", + "Kevin Cherepski ", + "Jose Tomas Tocino ", + "Adam Dobrawy ", + "Mikkel Munch Mortensen ", + "Andrzej Bistram ", + "Daniel Lipsitt ", +] + + +## ... source file abbreviated to get to loader examples ... + + + def use_model(self, model_name): + if self.include_models: + for model_pattern in self.include_models: + model_pattern = '^%s$' % model_pattern.replace('*', '.*') + if re.search(model_pattern, model_name): + return True + if self.exclude_models: + for model_pattern in self.exclude_models: + model_pattern = '^%s$' % model_pattern.replace('*', '.*') + if re.search(model_pattern, model_name): + return False + return not self.include_models + + def skip_field(self, field): + if self.exclude_columns: + if self.verbose_names and field.verbose_name: + if field.verbose_name in self.exclude_columns: + return True + if field.name in self.exclude_columns: + return True + return False + + +def generate_dot(graph_data, template='django_extensions/graph_models/digraph.dot'): + if isinstance(template, str): +~~ template = loader.get_template(template) + + if not isinstance(template, Template) and not (hasattr(template, 'template') and isinstance(template.template, Template)): + raise Exception("Default Django template loader isn't used. " + "This can lead to the incorrect template rendering. " + "Please, check the settings.") + + c = Context(graph_data).flatten() + dot = template.render(c) + + return dot + + +def generate_graph_data(*args, **kwargs): + generator = ModelGraph(*args, **kwargs) + generator.generate_graph_data() + return generator.get_graph_data() + + +def use_model(model, include_models, exclude_models): + generator = ModelGraph([], include_models=include_models, exclude_models=exclude_models) + return generator.use_model(model) + + + +## ... source file continues with no further loader examples... + +``` + + +## Example 3 from django-filter +[django-filter](https://github.com/carltongibson/django-filter) +([project documentation](https://django-filter.readthedocs.io/en/master/) +and +[PyPI page](https://pypi.org/project/django-filter/2.2.0/)) +makes it easier to filter down querysets from the +[Django ORM](/django-orm.html) by providing common bits of boilerplate +code. django-filter is provided as +[open source](https://github.com/carltongibson/django-filter/blob/master/LICENSE). + +[**django-filter / django_filters / rest_framework / backends.py**](https://github.com/carltongibson/django-filter/blob/master/django_filters/rest_framework/backends.py) + +```python +# backends.py +import warnings + +~~from django.template import loader +from django.utils.deprecation import RenameMethodsBase + +from .. import compat, utils +from . import filters, filterset + + +class RenameAttributes(utils.RenameAttributesBase, RenameMethodsBase): + renamed_attributes = ( + ('default_filter_set', 'filterset_base', utils.MigrationNotice), + ) + renamed_methods = ( + ('get_filter_class', 'get_filterset_class', utils.MigrationNotice), + ) + + +class DjangoFilterBackend(metaclass=RenameAttributes): + filterset_base = filterset.FilterSet + raise_exception = True + + @property + def template(self): + if compat.is_crispy(): + return 'django_filters/rest_framework/crispy_form.html' + return 'django_filters/rest_framework/form.html' + + +## ... source file abbreviated to get to loader examples ... + + + return AutoFilterSet + + return None + + def get_filterset_kwargs(self, request, queryset, view): + return { + 'data': request.query_params, + 'queryset': queryset, + 'request': request, + } + + def filter_queryset(self, request, queryset, view): + filterset = self.get_filterset(request, queryset, view) + if filterset is None: + return queryset + + if not filterset.is_valid() and self.raise_exception: + raise utils.translate_validation(filterset.errors) + return filterset.qs + + def to_html(self, request, queryset, view): + filterset = self.get_filterset(request, queryset, view) + if filterset is None: + return None + +~~ template = loader.get_template(self.template) + context = {'filter': filterset} + return template.render(context, request) + + def get_coreschema_field(self, field): + if isinstance(field, filters.NumberFilter): + field_cls = compat.coreschema.Number + else: + field_cls = compat.coreschema.String + return field_cls( + description=str(field.extra.get('help_text', '')) + ) + + def get_schema_fields(self, view): + assert compat.coreapi is not None, 'coreapi must be installed to use `get_schema_fields()`' + assert compat.coreschema is not None, 'coreschema must be installed to use `get_schema_fields()`' + + try: + queryset = view.get_queryset() + except Exception: + queryset = None + warnings.warn( + "{} is not compatible with schema generation".format(view.__class__) + ) + + + +## ... source file continues with no further loader examples... + +``` + + +## Example 4 from django-floppyforms +[django-floppyforms](https://github.com/jazzband/django-floppyforms) +([project documentation](https://django-floppyforms.readthedocs.io/en/latest/) +and +[PyPI page](https://pypi.org/project/django-floppyforms/)) +is a [Django](/django.html) code library for better control +over rendering HTML forms in your [templates](/template-engines.html). + +The django-floppyforms code is provided as +[open source](https://github.com/jazzband/django-floppyforms/blob/master/LICENSE) +and maintained by the collaborative developer community group +[Jazzband](https://jazzband.co/). + +[**django-floppyforms / floppyforms / widgets.py**](https://github.com/jazzband/django-floppyforms/blob/master/floppyforms/./widgets.py) + +```python +# widgets.py +import datetime +import re +from itertools import chain + +import django +from django import forms +from django.conf import settings +from django.forms.widgets import FILE_INPUT_CONTRADICTION +~~from django.template import loader +from django.utils import datetime_safe, formats +from django.utils.dates import MONTHS +from django.utils.encoding import force_str +from django.utils.html import conditional_escape +from django.utils.safestring import mark_safe +from django.utils.translation import gettext_lazy as _ + +from .compat import MULTIVALUE_DICT_TYPES, flatten_contexts + + +from django.forms.utils import to_current_timezone + + +RE_DATE = re.compile(r'(\d{4})-(\d\d?)-(\d\d?)$') + + +__all__ = ( + 'TextInput', 'PasswordInput', 'HiddenInput', 'ClearableFileInput', + 'FileInput', 'DateInput', 'DateTimeInput', 'TimeInput', 'Textarea', + 'CheckboxInput', 'Select', 'NullBooleanSelect', 'SelectMultiple', + 'RadioSelect', 'CheckboxSelectMultiple', 'SearchInput', 'RangeInput', + 'ColorInput', 'EmailInput', 'URLInput', 'PhoneNumberInput', 'NumberInput', + 'IPAddressInput', 'MultiWidget', 'Widget', 'SplitDateTimeWidget', + 'SplitHiddenDateTimeWidget', 'MultipleHiddenInput', 'SelectDateWidget', + + +## ... source file abbreviated to get to loader examples ... + + + + if value is None: + value = '' + + if value != '': + context['value'] = self.format_value(value) + + context.update(self.get_context_data()) + context['attrs'] = self.build_attrs(attrs) + + for key, attr in context['attrs'].items(): + if attr == 1: + if not isinstance(attr, bool): + context['attrs'][key] = str(attr) + + if self.datalist is not None: + context['datalist'] = self.datalist + return context + + def render(self, name, value, attrs=None, **kwargs): + template_name = kwargs.pop('template_name', None) + if template_name is None: + template_name = self.template_name + context = self.get_context(name, value, attrs=attrs or {}) + context = flatten_contexts(self.context_instance, context) +~~ return loader.render_to_string(template_name, context) + + +class TextInput(Input): + template_name = 'floppyforms/text.html' + input_type = 'text' + + def __init__(self, *args, **kwargs): + if kwargs.get('attrs', None) is not None: + self.input_type = kwargs['attrs'].pop('type', self.input_type) + super(TextInput, self).__init__(*args, **kwargs) + + +class PasswordInput(TextInput): + template_name = 'floppyforms/password.html' + input_type = 'password' + + def __init__(self, attrs=None, render_value=False): + super(PasswordInput, self).__init__(attrs) + self.render_value = render_value + + def render(self, name, value, attrs=None, renderer=None): + if not self.render_value: + value = None + return super(PasswordInput, self).render(name, value, attrs, renderer=renderer) + + +## ... source file abbreviated to get to loader examples ... + + + except ValueError: + pass + else: + match = RE_DATE.match(value) + if match: + year_val, month_val, day_val = map(int, match.groups()) + + context = self.get_context(name, value, attrs=attrs, + extra_context=extra_context) + + context['year_choices'] = [(i, i) for i in self.years] + context['year_val'] = year_val + + context['month_choices'] = list(MONTHS.items()) + context['month_val'] = month_val + + context['day_choices'] = [(i, i) for i in range(1, 32)] + context['day_val'] = day_val + + + if self.required is False: + context['year_choices'].insert(0, self.none_value) + context['month_choices'].insert(0, self.none_value) + context['day_choices'].insert(0, self.none_value) + +~~ return loader.render_to_string(self.template_name, context) + + def value_from_datadict(self, data, files, name): + y = data.get(self.year_field % name) + m = data.get(self.month_field % name) + d = data.get(self.day_field % name) + if y == m == d == "0": + return None + if y and m and d: + if settings.USE_L10N: + input_format = formats.get_format('DATE_INPUT_FORMATS')[0] + try: + date_value = datetime.date(int(y), int(m), int(d)) + except ValueError: + return '%s-%s-%s' % (y, m, d) + else: + date_value = datetime_safe.new_date(date_value) + return date_value.strftime(input_format) + else: + return '%s-%s-%s' % (y, m, d) + return data.get(name, None) + + + +## ... source file continues with no further loader examples... + +``` + + +## Example 5 from django-haystack +[django-haystack](https://github.com/django-haystack/django-haystack) +([project website](http://haystacksearch.org/) and +[PyPI page](https://pypi.org/project/django-haystack/)) +is a search abstraction layer that separates the Python search code +in a [Django](/django.html) web application from the search engine +implementation that it runs on, such as +[Apache Solr](http://lucene.apache.org/solr/), +[Elasticsearch](https://www.elastic.co/) +or [Whoosh](https://whoosh.readthedocs.io/en/latest/intro.html). + +The django-haystack project is open source under the +[BSD license](https://github.com/django-haystack/django-haystack/blob/master/LICENSE). + +[**django-haystack / haystack / fields.py**](https://github.com/django-haystack/django-haystack/blob/master/haystack/./fields.py) + +```python +# fields.py +import re +from inspect import ismethod + +~~from django.template import loader +from django.utils import datetime_safe + +from haystack.exceptions import SearchFieldError +from haystack.utils import get_model_ct_tuple + + +class NOT_PROVIDED: + pass + + +DATE_REGEX = re.compile( + r"^(?P\d{4})-(?P\d{2})-(?P\d{2})(?:|T00:00:00Z?)$" +) +DATETIME_REGEX = re.compile( + r"^(?P\d{4})-(?P\d{2})-(?P\d{2})(T|\s+)(?P\d{2}):(?P\d{2}):(?P\d{2}).*?$" +) + + + + +class SearchField(object): + + field_type = None + + + +## ... source file abbreviated to get to loader examples ... + + + return [] + + elif not hasattr(current_objects, "__iter__"): + current_objects = [current_objects] + + return current_objects + + def prepare_template(self, obj): + if self.instance_name is None and self.template_name is None: + raise SearchFieldError( + "This field requires either its instance_name variable to be populated or an explicit template_name in order to load the correct template." + ) + + if self.template_name is not None: + template_names = self.template_name + + if not isinstance(template_names, (list, tuple)): + template_names = [template_names] + else: + app_label, model_name = get_model_ct_tuple(obj) + template_names = [ + "search/indexes/%s/%s_%s.txt" + % (app_label, model_name, self.instance_name) + ] + +~~ t = loader.select_template(template_names) + return t.render({"object": obj}) + + def convert(self, value): + return value + + +class CharField(SearchField): + field_type = "string" + + def __init__(self, **kwargs): + if kwargs.get("facet_class") is None: + kwargs["facet_class"] = FacetCharField + + super(CharField, self).__init__(**kwargs) + + def prepare(self, obj): + return self.convert(super(CharField, self).prepare(obj)) + + def convert(self, value): + if value is None: + return None + + return str(value) + + + +## ... source file continues with no further loader examples... + +``` + + +## Example 6 from django-rest-framework +[Django REST Framework](https://github.com/encode/django-rest-framework) +([project homepage and documentation](https://www.django-rest-framework.org/), +[PyPI package information](https://pypi.org/project/djangorestframework/) +and [more resources on Full Stack Python](/django-rest-framework-drf.html)), +often abbreviated as "DRF", is a popular [Django](/django.html) extension +for building [web APIs](/application-programming-interfaces.html). +The project has fantastic documentation and a wonderful +[quickstart](https://www.django-rest-framework.org/tutorial/quickstart/) +that serve as examples of how to make it easier for newcomers +to get started. + +The project is open sourced under the +[Encode OSS Ltd. license](https://github.com/encode/django-rest-framework/blob/master/LICENSE.md). + +[**django-rest-framework / rest_framework / renderers.py**](https://github.com/encode/django-rest-framework/blob/master/rest_framework/./renderers.py) + +```python +# renderers.py +import base64 +from collections import OrderedDict +from urllib import parse + +from django import forms +from django.conf import settings +from django.core.exceptions import ImproperlyConfigured +from django.core.paginator import Page +from django.http.multipartparser import parse_header +~~from django.template import engines, loader +from django.urls import NoReverseMatch +from django.utils.html import mark_safe + +from rest_framework import VERSION, exceptions, serializers, status +from rest_framework.compat import ( + INDENT_SEPARATORS, LONG_SEPARATORS, SHORT_SEPARATORS, coreapi, coreschema, + pygments_css, yaml +) +from rest_framework.exceptions import ParseError +from rest_framework.request import is_form_media_type, override_method +from rest_framework.settings import api_settings +from rest_framework.utils import encoders, json +from rest_framework.utils.breadcrumbs import get_breadcrumbs +from rest_framework.utils.field_mapping import ClassLookupDict + + +def zero_as_none(value): + return None if value == 0 else value + + +class BaseRenderer: + media_type = None + format = None + charset = 'utf-8' + + +## ... source file abbreviated to get to loader examples ... + + + exception_template_names = [ + '%(status_code)s.html', + 'api_exception.html' + ] + charset = 'utf-8' + + def render(self, data, accepted_media_type=None, renderer_context=None): + renderer_context = renderer_context or {} + view = renderer_context['view'] + request = renderer_context['request'] + response = renderer_context['response'] + + if response.exception: + template = self.get_exception_template(response) + else: + template_names = self.get_template_names(response, view) + template = self.resolve_template(template_names) + + if hasattr(self, 'resolve_context'): + context = self.resolve_context(data, request, response) + else: + context = self.get_template_context(data, renderer_context) + return template.render(context, request=request) + + def resolve_template(self, template_names): +~~ return loader.select_template(template_names) + + def get_template_context(self, data, renderer_context): + response = renderer_context['response'] + if response.exception: + data['status_code'] = response.status_code + return data + + def get_template_names(self, response, view): + if response.template_name: + return [response.template_name] + elif self.template_name: + return [self.template_name] + elif hasattr(view, 'get_template_names'): + return view.get_template_names() + elif hasattr(view, 'template_name'): + return [view.template_name] + raise ImproperlyConfigured( + 'Returned a template response with no `template_name` attribute set on either the view or response' + ) + + def get_exception_template(self, response): + template_names = [name % {'status_code': response.status_code} + for name in self.exception_template_names] + + + +## ... source file abbreviated to get to loader examples ... + + + serializers.JSONField: { + 'base_template': 'textarea.html', + }, + }) + + def render_field(self, field, parent_style): + if isinstance(field._field, serializers.HiddenField): + return '' + + style = self.default_style[field].copy() + style.update(field.style) + if 'template_pack' not in style: + style['template_pack'] = parent_style.get('template_pack', self.template_pack) + style['renderer'] = self + + field = field.as_form_field() + + if style.get('input_type') == 'datetime-local' and isinstance(field.value, str): + field.value = field.value.rstrip('Z') + + if 'template' in style: + template_name = style['template'] + else: + template_name = style['template_pack'].strip('/') + '/' + style['base_template'] + +~~ template = loader.get_template(template_name) + context = {'field': field, 'style': style} + return template.render(context) + + def render(self, data, accepted_media_type=None, renderer_context=None): + renderer_context = renderer_context or {} + form = data.serializer + + style = renderer_context.get('style', {}) + if 'template_pack' not in style: + style['template_pack'] = self.template_pack + style['renderer'] = self + + template_pack = style['template_pack'].strip('/') + template_name = template_pack + '/' + self.base_template +~~ template = loader.get_template(template_name) + context = { + 'form': form, + 'style': style + } + return template.render(context) + + +class BrowsableAPIRenderer(BaseRenderer): + media_type = 'text/html' + format = 'api' + template = 'rest_framework/api.html' + filter_template = 'rest_framework/filters/base.html' + code_style = 'emacs' + charset = 'utf-8' + form_renderer_class = HTMLFormRenderer + + def get_default_renderer(self, view): + renderers = [renderer for renderer in view.renderer_classes + if not issubclass(renderer, BrowsableAPIRenderer)] + non_template_renderers = [renderer for renderer in renderers + if not hasattr(renderer, 'get_template_names')] + + if not renderers: + return None + + +## ... source file abbreviated to get to loader examples ... + + + if not hasattr(view, 'get_queryset') or not hasattr(view, 'filter_backends'): + return + + paginator = getattr(view, 'paginator', None) + if isinstance(data, list): + pass + elif paginator is not None and data is not None: + try: + paginator.get_results(data) + except (TypeError, KeyError): + return + elif not isinstance(data, list): + return + + queryset = view.get_queryset() + elements = [] + for backend in view.filter_backends: + if hasattr(backend, 'to_html'): + html = backend().to_html(request, queryset, view) + if html: + elements.append(html) + + if not elements: + return + +~~ template = loader.get_template(self.filter_template) + context = {'elements': elements} + return template.render(context) + + def get_context(self, data, accepted_media_type, renderer_context): + view = renderer_context['view'] + request = renderer_context['request'] + response = renderer_context['response'] + + renderer = self.get_default_renderer(view) + + raw_data_post_form = self.get_raw_data_form(data, view, 'POST', request) + raw_data_put_form = self.get_raw_data_form(data, view, 'PUT', request) + raw_data_patch_form = self.get_raw_data_form(data, view, 'PATCH', request) + raw_data_put_or_patch_form = raw_data_put_form or raw_data_patch_form + + response_headers = OrderedDict(sorted(response.items())) + renderer_content_type = '' + if renderer: + renderer_content_type = '%s' % renderer.media_type + if renderer.charset: + renderer_content_type += ' ;%s' % renderer.charset + response_headers['Content-Type'] = renderer_content_type + + if getattr(view, 'paginator', None) and view.paginator.display_page_controls: + + +## ... source file abbreviated to get to loader examples ... + + + 'put_form': self.get_rendered_html_form(data, view, 'PUT', request), + 'post_form': self.get_rendered_html_form(data, view, 'POST', request), + 'delete_form': self.get_rendered_html_form(data, view, 'DELETE', request), + 'options_form': self.get_rendered_html_form(data, view, 'OPTIONS', request), + + 'extra_actions': self.get_extra_actions(view, response.status_code), + + 'filter_form': self.get_filter_form(data, view, request), + + 'raw_data_put_form': raw_data_put_form, + 'raw_data_post_form': raw_data_post_form, + 'raw_data_patch_form': raw_data_patch_form, + 'raw_data_put_or_patch_form': raw_data_put_or_patch_form, + + 'display_edit_forms': bool(response.status_code != 403), + + 'api_settings': api_settings, + 'csrf_cookie_name': csrf_cookie_name, + 'csrf_header_name': csrf_header_name + } + + def render(self, data, accepted_media_type=None, renderer_context=None): + self.accepted_media_type = accepted_media_type or '' + self.renderer_context = renderer_context or {} + +~~ template = loader.get_template(self.template) + context = self.get_context(data, accepted_media_type, renderer_context) + ret = template.render(context, request=renderer_context['request']) + + response = renderer_context['response'] + if response.status_code == status.HTTP_204_NO_CONTENT: + response.status_code = status.HTTP_200_OK + + return ret + + +class AdminRenderer(BrowsableAPIRenderer): + template = 'rest_framework/admin.html' + format = 'admin' + + def render(self, data, accepted_media_type=None, renderer_context=None): + self.accepted_media_type = accepted_media_type or '' + self.renderer_context = renderer_context or {} + + response = renderer_context['response'] + request = renderer_context['request'] + view = self.renderer_context['view'] + + if response.status_code == status.HTTP_400_BAD_REQUEST: + self.error_form = self.get_rendered_html_form(data, view, request.method, request) + self.error_title = {'POST': 'Create', 'PUT': 'Edit'}.get(request.method, 'Errors') + + with override_method(view, request, 'GET') as request: + response = view.get(request, *view.args, **view.kwargs) + data = response.data + +~~ template = loader.get_template(self.template) + context = self.get_context(data, accepted_media_type, renderer_context) + ret = template.render(context, request=renderer_context['request']) + + if response.status_code == status.HTTP_201_CREATED and 'Location' in response: + response.status_code = status.HTTP_303_SEE_OTHER + response['Location'] = request.build_absolute_uri() + ret = '' + + if response.status_code == status.HTTP_204_NO_CONTENT: + response.status_code = status.HTTP_303_SEE_OTHER + try: + response['Location'] = self.get_breadcrumbs(request)[-2][1] + except KeyError: + response['Location'] = request.full_path + ret = '' + + return ret + + def get_context(self, data, accepted_media_type, renderer_context): + context = super().get_context( + data, accepted_media_type, renderer_context + ) + + paginator = getattr(context['view'], 'paginator', None) + + +## ... source file abbreviated to get to loader examples ... + + + except (KeyError, NoReverseMatch): + return + + +class DocumentationRenderer(BaseRenderer): + media_type = 'text/html' + format = 'html' + charset = 'utf-8' + template = 'rest_framework/docs/index.html' + error_template = 'rest_framework/docs/error.html' + code_style = 'emacs' + languages = ['shell', 'javascript', 'python'] + + def get_context(self, data, request): + return { + 'document': data, + 'langs': self.languages, + 'lang_htmls': ["rest_framework/docs/langs/%s.html" % language for language in self.languages], + 'lang_intro_htmls': ["rest_framework/docs/langs/%s-intro.html" % language for language in self.languages], + 'code_style': pygments_css(self.code_style), + 'request': request + } + + def render(self, data, accepted_media_type=None, renderer_context=None): + if isinstance(data, coreapi.Document): +~~ template = loader.get_template(self.template) + context = self.get_context(data, renderer_context['request']) + return template.render(context, request=renderer_context['request']) + else: +~~ template = loader.get_template(self.error_template) + context = { + "data": data, + "request": renderer_context['request'], + "response": renderer_context['response'], + "debug": settings.DEBUG, + } + return template.render(context, request=renderer_context['request']) + + +class SchemaJSRenderer(BaseRenderer): + media_type = 'application/javascript' + format = 'javascript' + charset = 'utf-8' + template = 'rest_framework/schema.js' + + def render(self, data, accepted_media_type=None, renderer_context=None): + codec = coreapi.codecs.CoreJSONCodec() + schema = base64.b64encode(codec.encode(data)).decode('ascii') + +~~ template = loader.get_template(self.template) + context = {'schema': mark_safe(schema)} + request = renderer_context['request'] + return template.render(context, request=request) + + +class MultiPartRenderer(BaseRenderer): + media_type = 'multipart/form-data; boundary=BoUnDaRyStRiNg' + format = 'multipart' + charset = 'utf-8' + BOUNDARY = 'BoUnDaRyStRiNg' + + def render(self, data, accepted_media_type=None, renderer_context=None): + from django.test.client import encode_multipart + + if hasattr(data, 'items'): + for key, value in data.items(): + assert not isinstance(value, dict), ( + "Test data contained a dictionary value for key '%s', " + "but multipart uploads do not support nested data. " + "You may want to consider using format='json' in this " + "test case." % key + ) + return encode_multipart(self.BOUNDARY, data) + + + +## ... source file continues with no further loader examples... + +``` + + +## Example 7 from django-request-token +[Django Request Token](https://github.com/yunojuno/django-request-token) +([PyPI package information](https://pypi.org/project/django-request-token/0.13/)) +encapsulates the logic for issuing expiring and one-time tokens +with a [Django](/django.html) web application to use with protected URLs. +Note that [PostgreSQL](/postgresql.html) as your backend +[database](/databases.html) is a dependency for using this project. + +The Django Request Token project is open sourced under the +[MIT license](https://github.com/yunojuno/django-request-token/blob/master/LICENSE). + +[**django-request-token / request_token / apps.py**](https://github.com/yunojuno/django-request-token/blob/master/request_token/./apps.py) + +```python +# apps.py +from __future__ import annotations + +from django.apps import AppConfig +from django.core.exceptions import ImproperlyConfigured +~~from django.template import TemplateDoesNotExist, loader + +from .settings import FOUR03_TEMPLATE + + +class RequestTokenAppConfig(AppConfig): + + name = "request_token" + verbose_name = "JWT Request Tokens" + + def ready(self) -> None: + super(RequestTokenAppConfig, self).ready() + if FOUR03_TEMPLATE: + check_template(FOUR03_TEMPLATE) + + +def check_template(template: str) -> None: + try: +~~ loader.get_template(template) + except TemplateDoesNotExist: + raise ImproperlyConfigured( + f"Custom request token template does not exist: '{template}'" + ) + + + +## ... source file continues with no further loader examples... + +``` + + +## Example 8 from graphite-web +[Graphite](https://github.com/graphite-project/graphite-web) +([project website](http://graphiteapp.org/), +[documentation](https://graphite.readthedocs.io/en/latest/) and +[PyPI package information](https://pypi.org/project/graphite-web/)) +is a metrics collection and visualization tool, built with both +Python and JavaScript. Metrics are collected by a Node.js application +and displayed using a [Django](/django.html) web application, +called "Graphite-Web", which is one of three core projects under +the Graphite umbrella (the other two are +[Carbon](https://github.com/graphite-project/carbon) and +[Whisper](https://github.com/graphite-project/whisper)). + +Graphite is provided as open sourced under the +[Apache License 2.0](https://github.com/graphite-project/whisper/blob/master/LICENSE). + +[**graphite-web / webapp / graphite / views.py**](https://github.com/graphite-project/graphite-web/blob/master/webapp/graphite/views.py) + +```python +# views.py +import traceback +from django.http import HttpResponseServerError +~~from django.template import loader + + +def server_error(request, template_name='500.html'): +~~ template = loader.get_template(template_name) + context = {'stacktrace' : traceback.format_exc()} + return HttpResponseServerError(template.render(context)) + + + +## ... source file continues with no further loader examples... + +``` + diff --git a/content/pages/examples/django/django-template-loaders-filesystem-loader.markdown b/content/pages/examples/django/django-template-loaders-filesystem-loader.markdown new file mode 100644 index 000000000..88ec1cc4a --- /dev/null +++ b/content/pages/examples/django/django-template-loaders-filesystem-loader.markdown @@ -0,0 +1,59 @@ +title: django.template.loaders.filesystem Loader Example Code +category: page +slug: django-template-loaders-filesystem-loader-examples +sortorder: 500011398 +toc: False +sidebartitle: django.template.loaders.filesystem Loader +meta: Example code for understanding how to use the Loader class from the django.template.loaders.filesystem module of the Django project. + + +`Loader` is a class within the `django.template.loaders.filesystem` module of the Django project. + + + +## Example 1 from django-markdown-view +[django-markdown-view](https://github.com/rgs258/django-markdown-view) +([PyPI package information](https://pypi.org/project/django-markdown-view/)) +is a Django extension for serving [Markdown](/markdown.html) files as +[Django templates](/django-templates.html). The project is open +sourced under the +[BSD 3-Clause "New" or "Revised" license](https://github.com/rgs258/django-markdown-view/blob/master/LICENSE). + +[**django-markdown-view / markdown_view / loaders.py**](https://github.com/rgs258/django-markdown-view/blob/master/markdown_view/./loaders.py) + +```python +# loaders.py +from django.conf import settings +from django.core.exceptions import SuspiciousFileOperation +from django.template import Origin +~~from django.template.loaders.filesystem import Loader as FilesystemLoader +from django.template.utils import get_app_template_dirs +from django.utils._os import safe_join + +from markdown_view.constants import DEFAULT_MARKDOWN_VIEW_LOADER_TEMPLATES_DIR + + +class MarkdownLoader(FilesystemLoader): + + def get_dirs(self): + base_dir = getattr( + settings, + "MARKDOWN_VIEW_BASE_DIR", + getattr( + settings, + "BASE_DIR", + None) + ) + dirs = [*get_app_template_dirs( + getattr( + settings, + "MARKDOWN_VIEW_LOADER_TEMPLATES_DIR", + DEFAULT_MARKDOWN_VIEW_LOADER_TEMPLATES_DIR + ) + )] + + +## ... source file continues with no further Loader examples... + +``` + diff --git a/content/pages/examples/flask/flask-app-badrequest.markdown b/content/pages/examples/flask/flask-app-badrequest.markdown index 656406cf3..a2c3c3abd 100644 --- a/content/pages/examples/flask/flask-app-badrequest.markdown +++ b/content/pages/examples/flask/flask-app-badrequest.markdown @@ -20,7 +20,7 @@ that accepts POSTs. and ImmutableDict are several other callables with code examples from the same `flask.app` package. -These topics are also useful while reading the `BadRequest` examples: +These subjects go along with the `BadRequest` code examples: * [web development](/web-development.html) and [web design](/web-design.html) * [Flask](/flask.html) and [web framework](/web-frameworks.html) concepts @@ -90,7 +90,6 @@ from ..const import ( ## ... source file abbreviated to get to BadRequest examples ... - API_SELECT_COLUMNS_RIS_KEY, API_SHOW_COLUMNS_RES_KEY, API_SHOW_COLUMNS_RIS_KEY, API_SHOW_TITLE_RES_KEY, @@ -99,6 +98,7 @@ from ..const import ( PERMISSION_PREFIX, ) from ..exceptions import FABException, InvalidOrderByColumnFABException +from ..hooks import get_before_request_hooks, wrap_route_handler_with_hooks from ..security.decorators import permission_name, protect log = logging.getLogger(__name__) @@ -149,7 +149,7 @@ def rison(schema=None): ## Example 2 from Flask-WTF [Flask-WTF](https://github.com/lepture/flask-wtf) -([project documentation](https://flask-wtf.readthedocs.io/en/stable/) +([project documentation](https://flask-wtf.readthedocs.io/) and [PyPI page](https://pypi.org/project/Flask-WTF/)) provides a bridge between [Flask](/flask.html) and the the @@ -166,6 +166,7 @@ import hashlib import logging import os import warnings +from urllib.parse import urlparse from functools import wraps from flask import Blueprint, current_app, g, request, session @@ -175,7 +176,7 @@ from werkzeug.security import safe_str_cmp from wtforms import ValidationError from wtforms.csrf.core import CSRF -from ._compat import FlaskWTFDeprecationWarning, string_types, urlparse +from ._compat import FlaskWTFDeprecationWarning __all__ = ('generate_csrf', 'validate_csrf', 'CSRFProtect') logger = logging.getLogger(__name__) @@ -222,7 +223,7 @@ class CsrfProtect(CSRFProtect): '"flask_wtf.CsrfProtect" has been renamed to "CSRFProtect" ' 'and will be removed in 1.0.' ), stacklevel=2) - super(CsrfProtect, self).__init__(app=app) + super().__init__(app=app) ~~class CSRFError(BadRequest): @@ -265,22 +266,21 @@ The code is open sourced under the ~~from werkzeug.exceptions import BadRequest, Forbidden, HTTPException, NotFound from indico.util.i18n import _ -from indico.util.string import to_unicode def get_error_description(exception): try: description = exception.description except AttributeError: - return to_unicode(exception.message) + return str(exception) if isinstance(exception, Forbidden) and description == Forbidden.description: - return _(u"You are not allowed to access this page.") + return _('You are not allowed to access this page.') elif isinstance(exception, NotFound) and description == NotFound.description: - return _(u"The page you are looking for doesn't exist.") + return _("The page you are looking for doesn't exist.") ~~ elif isinstance(exception, BadRequest) and description == BadRequest.description: - return _(u"The request was invalid or contained invalid arguments.") + return _('The request was invalid or contained invalid arguments.') else: - return to_unicode(description) + return str(description) class IndicoError(Exception): diff --git a/content/pages/examples/flask/flask-app-flask.markdown b/content/pages/examples/flask/flask-app-flask.markdown index 333598654..0702a72ad 100644 --- a/content/pages/examples/flask/flask-app-flask.markdown +++ b/content/pages/examples/flask/flask-app-flask.markdown @@ -88,35 +88,102 @@ as-is to run CTF events, or modified for custom rules for related scenarios. CTFd is open sourced under the [Apache License 2.0](https://github.com/CTFd/CTFd/blob/master/LICENSE). -[**CTFd / manage.py**](https://github.com/CTFd/CTFd/blob/master/././manage.py) +[**CTFd / tests / test_themes.py**](https://github.com/CTFd/CTFd/blob/master/./tests/test_themes.py) ```python -# manage.py -~~from flask import Flask -from flask_sqlalchemy import SQLAlchemy -from flask_script import Manager -from flask_migrate import Migrate, MigrateCommand -from CTFd import create_app -from CTFd.utils import get_config as get_config_util, set_config as set_config_util -from CTFd.models import * +# test_themes.py + +import os +import shutil + +import pytest +from flask import render_template, render_template_string, request +from jinja2.exceptions import TemplateNotFound +from jinja2.sandbox import SecurityError +from werkzeug.test import Client -app = create_app() +from CTFd.config import TestingConfig +from CTFd.utils import get_config, set_config +from tests.helpers import create_ctfd, destroy_ctfd, gen_user, login_as_user -manager = Manager(app) -manager.add_command("db", MigrateCommand) +def test_themes_run_in_sandbox(): + app = create_ctfd() + with app.app_context(): + try: + app.jinja_env.from_string( + "{{ ().__class__.__bases__[0].__subclasses__()[40]('./test_utils.py').read() }}" + ).render() + except SecurityError: + pass + except Exception as e: + raise e + destroy_ctfd(app) + + +def test_themes_cant_access_configpy_attributes(): + app = create_ctfd() + with app.app_context(): + assert app.config["SECRET_KEY"] == "AAAAAAAAAAAAAAAAAAAA" + assert ( + app.jinja_env.from_string("{{ get_config('SECRET_KEY') }}").render() + + +## ... source file abbreviated to get to Flask examples ... -def jsenums(): - from CTFd.constants import JS_ENUMS - import json - import os - path = os.path.join(app.root_path, "themes/core/assets/js/constants.js") - with open(path, "w+") as f: - for k, v in JS_ENUMS.items(): - f.write("const {} = Object.freeze({});".format(k, json.dumps(v))) + r = client.get("/challenges") + assert r.status_code == 200 + assert "Challenges" in r.get_data(as_text=True) + r = client.get("/scoreboard") + assert r.status_code == 200 + assert "Scoreboard" in r.get_data(as_text=True) + destroy_ctfd(app) + + +def test_that_request_path_hijacking_works_properly(): + app = create_ctfd(setup=False, application_root="/ctf") + assert app.request_class.__name__ == "CTFdRequest" + with app.app_context(): + with app.test_request_context("/challenges"): + assert request.path == "/ctf/challenges" + destroy_ctfd(app) + + app = create_ctfd() + assert app.request_class.__name__ == "CTFdRequest" + with app.app_context(): + with app.test_request_context("/challenges"): + assert request.path == "/challenges" + +~~ from flask import Flask + +~~ test_app = Flask("test") + assert test_app.request_class.__name__ == "Request" + with test_app.test_request_context("/challenges"): + assert request.path == "/challenges" + destroy_ctfd(app) + + +def test_theme_fallback_config(): + + class ThemeFallbackConfig(TestingConfig): + THEME_FALLBACK = False + + app = create_ctfd(config=ThemeFallbackConfig) + try: + os.mkdir(os.path.join(app.root_path, "themes", "foo_fallback")) + except OSError: + pass + + with app.app_context(): + app.config["THEME_FALLBACK"] = False + set_config("ctf_theme", "foo_fallback") + assert app.config["THEME_FALLBACK"] == False + with app.test_client() as client: + try: + r = client.get("/") ## ... source file continues with no further Flask examples... @@ -137,48 +204,158 @@ forms, and internationalization support. Flask App Builder is provided under the [BSD 3-Clause "New" or "Revised" license](https://github.com/dpgaspar/Flask-AppBuilder/blob/master/LICENSE). -[**Flask AppBuilder / flask_appbuilder / tests / _test_oauth_registration_role.py**](https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/tests/_test_oauth_registration_role.py) +[**Flask AppBuilder / flask_appbuilder / tests / test_fab_cli.py**](https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/tests/test_fab_cli.py) ```python -# _test_oauth_registration_role.py +# test_fab_cli.py +import glob +import json import logging -import unittest +import os +import tempfile +from click.testing import CliRunner ~~from flask import Flask from flask_appbuilder import AppBuilder, SQLA +from flask_appbuilder.cli import ( + create_app, + create_permissions, + create_user, + export_roles, + import_roles, + list_users, + list_views, + reset_password, +) +from .base import FABTestCase logging.basicConfig(format="%(asctime)s:%(levelname)s:%(name)s:%(message)s") logging.getLogger().setLevel(logging.DEBUG) log = logging.getLogger(__name__) +APP_DIR = "myapp" -class OAuthRegistrationRoleTestCase(unittest.TestCase): + +class FlaskTestCase(FABTestCase): def setUp(self): -~~ self.app = Flask(__name__) - self.app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False - self.db = SQLA(self.app) + pass + - def tearDown(self): - self.appbuilder = None - self.app = None - self.db = None +## ... source file abbreviated to get to Flask examples ... - def test_self_registration_not_enabled(self): - self.app.config["AUTH_USER_REGISTRATION"] = False - self.appbuilder = AppBuilder(self.app, self.db.session) - result = self.appbuilder.sm.auth_user_oauth(userinfo={"username": "testuser"}) + self.assertIn("User bob created.", result.output) - self.assertIsNone(result) - self.assertEqual(len(self.appbuilder.sm.get_all_users()), 0) + result = runner.invoke(list_users, []) + self.assertIn("bob", result.output) - def test_register_and_attach_static_role(self): - self.app.config["AUTH_USER_REGISTRATION"] = True - self.app.config["AUTH_USER_REGISTRATION_ROLE"] = "Public" - self.appbuilder = AppBuilder(self.app, self.db.session) + runner.invoke(create_permissions, []) - user = self.appbuilder.sm.auth_user_oauth(userinfo={"username": "testuser"}) + runner.invoke(reset_password, ["--username=bob", "--password=bar"]) + + def test_list_views(self): + os.environ["FLASK_APP"] = "app:app" + runner = CliRunner() + with runner.isolated_filesystem(): + result = runner.invoke(list_views, []) + self.assertIn("List of registered views", result.output) + self.assertIn(" Route:/api/v1/security", result.output) + + +class SQLAlchemyImportExportTestCase(FABTestCase): + def setUp(self): + with open("flask_appbuilder/tests/data/roles.json", "r") as fd: + self.expected_roles = json.loads(fd.read()) + + def test_export_roles(self): + with tempfile.TemporaryDirectory() as tmp_dir: +~~ app = Flask("src_app") + app.config.from_object("flask_appbuilder.tests.config_security") + app.config[ + "SQLALCHEMY_DATABASE_URI" + ] = f"sqlite:///{os.path.join(tmp_dir, 'src.db')}" + db = SQLA(app) + app_builder = AppBuilder(app, db.session) # noqa: F841 + cli_runner = app.test_cli_runner() + + path = os.path.join(tmp_dir, "roles.json") + + export_result = cli_runner.invoke(export_roles, [f"--path={path}"]) + + self.assertEqual(export_result.exit_code, 0) + self.assertTrue(os.path.exists(path)) + + with open(path, "r") as fd: + resulting_roles = json.loads(fd.read()) + + for expected_role in self.expected_roles: + match = [ + r for r in resulting_roles if r["name"] == expected_role["name"] + ] + self.assertTrue(match) + resulting_role = match[0] + resulting_role_permission_view_menus = { + (pvm["permission"]["name"], pvm["view_menu"]["name"]) + for pvm in resulting_role["permissions"] + } + expected_role_permission_view_menus = { + (pvm["permission"]["name"], pvm["view_menu"]["name"]) + for pvm in expected_role["permissions"] + } + self.assertEqual( + resulting_role_permission_view_menus, + expected_role_permission_view_menus, + ) + + def test_export_roles_filename(self): + with tempfile.TemporaryDirectory() as tmp_dir: +~~ app = Flask("src_app") + app.config.from_object("flask_appbuilder.tests.config_security") + + app.config[ + "SQLALCHEMY_DATABASE_URI" + ] = f"sqlite:///{os.path.join(tmp_dir, 'src.db')}" + db = SQLA(app) + app_builder = AppBuilder(app, db.session) # noqa: F841 + + owd = os.getcwd() + os.chdir(tmp_dir) + cli_runner = app.test_cli_runner() + export_result = cli_runner.invoke(export_roles) + os.chdir(owd) + + self.assertEqual(export_result.exit_code, 0) + self.assertGreater( + len(glob.glob(os.path.join(tmp_dir, "roles_export_*"))), 0 + ) + + def test_import_roles(self): + with tempfile.TemporaryDirectory() as tmp_dir: +~~ app = Flask("dst_app") + app.config[ + "SQLALCHEMY_DATABASE_URI" + ] = f"sqlite:///{os.path.join(tmp_dir, 'dst.db')}" + db = SQLA(app) + app_builder = AppBuilder(app, db.session) + cli_runner = app.test_cli_runner() + + path = os.path.join(tmp_dir, "roles.json") + + with open(path, "w") as fd: + fd.write(json.dumps(self.expected_roles)) + + self.assertEqual(len(app_builder.sm.get_all_roles()), 2) + + import_result = cli_runner.invoke(import_roles, [f"--path={path}"]) + self.assertEqual(import_result.exit_code, 0) + + resulting_roles = app_builder.sm.get_all_roles() + + for expected_role in self.expected_roles: + match = [r for r in resulting_roles if r.name == expected_role["name"]] + self.assertTrue(match) + resulting_role = match[0] @@ -215,7 +392,6 @@ from sqlalchemy import event from sqlalchemy.engine import Engine from sqlalchemy.exc import OperationalError, ProgrammingError -from flaskbb._compat import iteritems, string_types from flaskbb.extensions import (alembic, allows, babel, cache, celery, csrf, db, debugtoolbar, limiter, login_manager, mail, redis_store, themes, whooshee) @@ -225,7 +401,7 @@ from flaskbb.plugins.models import PluginRegistry from flaskbb.plugins.utils import remove_zombie_plugins_from_db, template_hook from flaskbb.user.models import Guest, User from flaskbb.utils.helpers import (app_config_from_env, crop_title, - format_date, format_datetime, + format_date, format_time, format_datetime, forum_is_unread, get_alembic_locations, get_flaskbb_config, is_online, mark_online, render_template, time_since, time_utcnow, @@ -249,6 +425,7 @@ from .forum import views as forum_views # noqa from .management import views as management_views # noqa from .user import views as user_views # noqa + logger = logging.getLogger(__name__) @@ -296,7 +473,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) backend, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling and many others. flask-base is provided as open source under the @@ -593,7 +770,7 @@ class HTTPAuthTestCase(unittest.TestCase): is an example application that ties together the [intTellInput.js](https://github.com/jackocnr/intl-tel-input) JavaScript plugin with the -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) form-handling +[Flask-WTF](https://flask-wtf.readthedocs.io/) form-handling library. flask-phone-input is provided as open source under the [MIT license](https://github.com/miguelgrinberg/flask-phone-input/blob/1a1c227c044474ce0fe133493d7f8b0fb8312409/LICENSE). @@ -645,7 +822,7 @@ def index(): starter project to build a software-as-a-service (SaaS) web application in [Flask](/flask.html), with [Stripe](/stripe.html) for billing. The boilerplate relies on many common Flask extensions such as -[Flask-WTF](https://flask-wtf.readthedocs.io/en/latest/), +[Flask-WTF](https://flask-wtf.readthedocs.io/), [Flask-Login](https://flask-login.readthedocs.io/en/latest/), [Flask-Admin](https://flask-admin.readthedocs.io/en/latest/), and many others. The project is provided as open source under the @@ -689,7 +866,83 @@ from app.models import User ``` -## Example 12 from Flask-SocketIO +## Example 12 from Flask-Security-Too +[Flask-Security-Too](https://github.com/Flask-Middleware/flask-security/) +([PyPi page](https://pypi.org/project/Flask-Security-Too/) and +[project documentation](https://flask-security-too.readthedocs.io/en/stable/)) +is a maintained fork of the original +[Flask-Security](https://github.com/mattupstate/flask-security) project that +makes it easier to add common security features to [Flask](/flask.html) +web applications. A few of the critical goals of the Flask-Security-Too +project are ensuring JavaScript client-based single-page applications (SPAs) +can work securely with Flask-based backends and that guidance by the +[OWASP](https://owasp.org/) organization is followed by default. + +The Flask-Security-Too project is provided as open source under the +[MIT license](https://github.com/Flask-Middleware/flask-security/blob/master/LICENSE). + +[**Flask-Security-Too / flask_security / utils.py**](https://github.com/Flask-Middleware/flask-security/blob/master/flask_security/./utils.py) + +```python +# utils.py + flash, + g, + request, + render_template, + session, + url_for, +) +from flask.json import JSONEncoder +from flask_login import login_user as _login_user +from flask_login import logout_user as _logout_user +from flask_login import current_user +from flask_login import COOKIE_NAME as REMEMBER_COOKIE_NAME +from flask_principal import AnonymousIdentity, Identity, identity_changed, Need +from flask_wtf import csrf +from wtforms import ValidationError +from itsdangerous import BadSignature, SignatureExpired +from werkzeug import __version__ as werkzeug_version +from werkzeug.local import LocalProxy +from werkzeug.datastructures import MultiDict + +from .quart_compat import best, get_quart_status +from .proxies import _security, _datastore, _pwd_context, _hashing_context +from .signals import user_authenticated + +if t.TYPE_CHECKING: # pragma: no cover +~~ from flask import Flask, Response + from .datastore import User + +SB = t.Union[str, bytes] + + +localize_callback = LocalProxy(lambda: _security.i18n_domain.gettext) + +FsPermNeed = partial(Need, "fsperm") +FsPermNeed.__doc__ = """A need with the method preset to `"fsperm"`.""" + + +def _(translate): + return translate + + +def get_request_attr(name: str) -> t.Any: + return getattr(_request_ctx_stack.top, name, None) + + +def set_request_attr(name, value): + return setattr(_request_ctx_stack.top, name, value) + + +if get_quart_status(): # pragma: no cover + + +## ... source file continues with no further Flask examples... + +``` + + +## Example 13 from Flask-SocketIO [Flask-SocketIO](https://github.com/miguelgrinberg/Flask-SocketIO) ([PyPI package information](https://pypi.org/project/Flask-SocketIO/), [official tutorial](https://blog.miguelgrinberg.com/post/easy-websockets-with-flask-and-gevent) @@ -740,8 +993,8 @@ def on_disconnect(): disconnected = '/' -@socketio.on('connect', namespace='/test') -def on_connect_test(): +@socketio.event(namespace='/test') +def connect(): send('connected-test') @@ -789,10 +1042,15 @@ def on_connect_test(): self.assertEqual(len(received), 1) self.assertEqual(received[0]['args'], {'connected': 'foo'}) - -if __name__ == '__main__': - unittest.main() - + def test_encode_decode(self): + client = socketio.test_client(app) + client.get_received() + data = {'foo': 'bar', 'invalid': socketio} + self.assertRaises(TypeError, client.emit, 'my custom event', data, + callback=True) + data = {'foo': 'bar'} + ack = client.emit('my custom event', data, callback=True) + data['foo'] = 'baz' ## ... source file continues with no further Flask examples... @@ -800,7 +1058,7 @@ if __name__ == '__main__': ``` -## Example 13 from Flask-User +## Example 14 from Flask-User [Flask-User](https://github.com/lingthio/Flask-User) ([PyPI information](https://pypi.org/project/Flask-User/) and @@ -883,7 +1141,7 @@ class UserManager(UserManager__Settings, UserManager__Utils, UserManager__Views) ``` -## Example 14 from Flask-VueJs-Template +## Example 15 from Flask-VueJs-Template [Flask-VueJs-Template](https://github.com/gtalarico/flask-vuejs-template) ([demo site](https://flask-vuejs-template.herokuapp.com/)) is a minimal [Flask](/flask.html) boilerplate starter project that @@ -924,7 +1182,7 @@ def index_client(): ``` -## Example 15 from Flasky +## Example 16 from Flasky [Flasky](https://github.com/miguelgrinberg/flasky) is a wonderful example application by [Miguel Grinberg](https://github.com/miguelgrinberg) that he builds @@ -988,7 +1246,7 @@ def create_app(config_name): ``` -## Example 16 from Datadog Flask Example App +## Example 17 from Datadog Flask Example App The [Datadog Flask example app](https://github.com/DataDog/trace-examples/tree/master/python/flask) contains many examples of the [Flask](/flask.html) core functions available to a developer using the [web framework](/web-frameworks.html). @@ -1049,7 +1307,7 @@ def before_request(): ``` -## Example 17 from keras-flask-deploy-webapp +## Example 18 from keras-flask-deploy-webapp The [keras-flask-deploy-webapp](https://github.com/mtobeiyf/keras-flask-deploy-webapp) project combines the [Flask](/flask.html) [web framework](/web-frameworks.html) @@ -1084,7 +1342,8 @@ from util import base64_to_pil ~~app = Flask(__name__) -from keras.applications.mobilenet_v2 import MobileNetV2 + +from tensorflow.keras.applications.mobilenet_v2 import MobileNetV2 model = MobileNetV2(weights='imagenet') print('Model loaded. Check http://127.0.0.1:5000/') @@ -1107,13 +1366,12 @@ def model_predict(img, model): - ## ... source file continues with no further Flask examples... ``` -## Example 18 from sandman2 +## Example 19 from sandman2 [sandman2](https://github.com/jeffknupp/sandman2) ([project documentation](https://sandman2.readthedocs.io/en/latest/) and @@ -1194,7 +1452,7 @@ def get_app( ``` -## Example 19 from Science Flask +## Example 20 from Science Flask [Science Flask](https://github.com/danielhomola/science_flask) is a [Flask](/flask.html)-powered web application for online scientific research tools. The project was built as a template @@ -1263,7 +1521,80 @@ def create_celery_app(): ``` -## Example 20 from tedivms-flask +## Example 21 from ShortMe +[ShortMe](https://github.com/AcrobaticPanicc/ShortMe-URL-Shortener) +is a [Flask](/flask.html) app that creates a shortened URL +that redirects to another, typically much longer, URL. The +project is provided as open source under the +[MIT license](https://github.com/AcrobaticPanicc/ShortMe-URL-Shortener/blob/main/LICENSE). + +[**ShortMe / app / setup / setup.py**](https://github.com/AcrobaticPanicc/ShortMe-URL-Shortener/blob/main/app/setup/setup.py) + +```python +# setup.py +import os + +~~from flask import Flask +from flask_restful import Api +from dotenv import load_dotenv + +from app.server.db.extensions import db +from app.server.db.models import AuthToken +from app.server.routes.index import index_blueprint +from app.server.routes.internal.redirect_to_url import redirect_to_url_blueprint +from app.server.routes.internal.favicon import app_blueprint +from app.server.routes.internal.send_verification_code import send_otp_blueprint +from app.server.routes.internal.shorten_url import shorten_url_blueprint +from app.server.routes.your_short_url import your_short_url_blueprint +from app.server.routes.total_clicks import total_clicks_blueprint +from app.server.routes.error import error_blueprint +from app.server.routes.page_not_found import page_not_found_blueprint +from app.server.routes.api_doc import api_doc_blueprint +from app.server.routes.get_token import get_token_blueprint +from app.server.routes.your_api_token import your_api_token_blueprint +from app.server.routes.verify_code import verify_code_blueprint + +from app.server.api.api import Shorten, TotalClicks, GetToken + + +def create_app(config_file): + app_path = os.path.dirname(os.path.abspath(__file__)) + project_folder = os.path.expanduser(app_path) + load_dotenv(os.path.join(project_folder, '.env')) + +~~ app = Flask(__name__, template_folder='../client/templates', static_folder='../client/static') + api = Api(app) + app.config.from_pyfile(config_file) + + db.init_app(app) + + with app.app_context(): + db.drop_all() + db.create_all() + + app_auth_token = app.secret_key + auth_token = AuthToken(auth_token=app_auth_token) + db.session.add(auth_token) + db.session.commit() + + api.add_resource(Shorten, '/api/shorten') + api.add_resource(GetToken, '/api/get_token') + api.add_resource(TotalClicks, '/api/total_clicks') + + app.register_blueprint(index_blueprint) + app.register_blueprint(page_not_found_blueprint) + app.register_blueprint(redirect_to_url_blueprint) + app.register_blueprint(your_short_url_blueprint) + app.register_blueprint(total_clicks_blueprint) + app.register_blueprint(error_blueprint) + + +## ... source file continues with no further Flask examples... + +``` + + +## Example 22 from tedivms-flask [tedivm's flask starter app](https://github.com/tedivm/tedivms-flask) is a base of [Flask](/flask.html) code and related projects such as [Celery](/celery.html) which provides a template to start your own @@ -1286,7 +1617,8 @@ import os import requests import yaml -~~from flask import Flask, session, render_template +~~from flask import Flask, render_template +from flask import session as current_session from flask_mail import Mail from flask_migrate import Migrate, MigrateCommand from flask.sessions import SessionInterface @@ -1392,7 +1724,7 @@ def create_app(extra_config_settings={}): ``` -## Example 21 from trape +## Example 23 from trape [trape](https://github.com/jofpin/trape) is a research tool for tracking people's activities that are logged digitally. The tool uses [Flask](/flask.html) to create a web front end to view aggregated data @@ -1414,6 +1746,8 @@ from core.db import Database import os import sys import platform +import urllib +import requests from multiprocessing import Process trape = core.stats.trape @@ -1424,13 +1758,11 @@ db = Database() class victim_server(object): @app.route("/" + trape.victim_path) def homeVictim(): - opener = urllib2.build_opener() - headers = victim_headers(request.user_agent) - opener.addheaders = headers + r = requests.get(trape.url_to_clone, headers=victim_headers2(request.user_agent)) if (trape.type_lure == 'local'): html = assignScripts(victim_inject_code(render_template("/" + trape.url_to_clone), 'payload', '/', trape.gmaps, trape.ipinfo)) else: - html = assignScripts(victim_inject_code(opener.open(trape.url_to_clone).read(), 'payload', trape.url_to_clone, trape.gmaps, trape.ipinfo)) + html = assignScripts(victim_inject_code(r.content, 'payload', trape.url_to_clone, trape.gmaps, trape.ipinfo)) ## ... source file continues with no further Flask examples... diff --git a/content/pages/examples/flask/flask-app-headers.markdown b/content/pages/examples/flask/flask-app-headers.markdown index 79b73c775..e279fec3c 100644 --- a/content/pages/examples/flask/flask-app-headers.markdown +++ b/content/pages/examples/flask/flask-app-headers.markdown @@ -22,7 +22,7 @@ Flask web applications. and ImmutableDict are several other callables with code examples from the same `flask.app` package. -You should read up on these subjects along with these `Headers` examples: +These topics are also useful while reading the `Headers` examples: * [web development](/web-development.html) and [web design](/web-design.html) * [Flask](/flask.html) and [web framework](/web-frameworks.html) concepts @@ -47,10 +47,12 @@ import random import string import uuid from collections import namedtuple +from contextlib import contextmanager from unittest.mock import Mock, patch import requests from flask.testing import FlaskClient +from freezegun import freeze_time from sqlalchemy.engine.url import make_url from sqlalchemy_utils import drop_database ~~from werkzeug.datastructures import Headers @@ -60,23 +62,33 @@ from CTFd.cache import cache, clear_standings from CTFd.config import TestingConfig from CTFd.models import ( Awards, + ChallengeComments, ChallengeFiles, Challenges, + ChallengeTopics, + Comments, Fails, + Fields, Files, Flags, Hints, Notifications, + PageComments, PageFiles, Pages, Solves, Tags, + TeamComments, Teams, Tokens, + Topics, Tracking, Unlocks, + UserComments, Users, ) +from CTFd.utils import set_config +from tests.constants.time import FreezeTimes text_type = str binary_type = bytes @@ -98,25 +110,25 @@ class CTFdTestClient(FlaskClient): return super(CTFdTestClient, self).open(*args, **kwargs) -def create_ctfd( - ctf_name="CTFd", - ctf_description="CTF description", - name="admin", - email="admin@ctfd.io", - password="password", - user_mode="users", - setup=True, - enable_plugins=False, - application_root="/", - config=TestingConfig, -): - if enable_plugins: - config.SAFE_MODE = False - else: - config.SAFE_MODE = True +class ctftime: + @contextmanager + def init(): + try: + set_config("start", FreezeTimes.START) + set_config("end", FreezeTimes.END) + yield + finally: + set_config("start", None) + set_config("end", None) - config.APPLICATION_ROOT = application_root - url = make_url(config.SQLALCHEMY_DATABASE_URI) + @contextmanager + def not_started(): + try: + freezer = freeze_time(FreezeTimes.NOT_STARTED) + frozen_time = freezer.start() + yield frozen_time + finally: + freezer.stop() ## ... source file continues with no further Headers examples... @@ -139,8 +151,6 @@ Flask RESTX is provided as open source under the ```python # api.py -from __future__ import unicode_literals - import difflib import inspect from itertools import chain @@ -149,6 +159,7 @@ import operator import re import six import sys +import warnings from collections import OrderedDict from functools import wraps, partial @@ -156,7 +167,10 @@ from types import MethodType from flask import url_for, request, current_app from flask import make_response as original_flask_make_response -from flask.helpers import _endpoint_from_view_func +try: + from flask.helpers import _endpoint_from_view_func +except ImportError: + from flask.scaffold import _endpoint_from_view_func from flask.signals import got_request_exception from jsonschema import RefResolver @@ -170,7 +184,13 @@ from werkzeug.exceptions import ( NotAcceptable, InternalServerError, ) -from werkzeug.wrappers import BaseResponse + +from werkzeug import __version__ as werkzeug_version + +if werkzeug_version.split('.')[0] >= '2': + from werkzeug.wrappers import Response as BaseResponse +else: + from werkzeug.wrappers import BaseResponse from . import apidoc from .mask import ParseError, MaskError @@ -182,28 +202,22 @@ from .utils import default_id, camel_to_dash, unpack from .representations import output_json from ._http import HTTPStatus -RE_RULES = re.compile("(<.*>)") - -HEADERS_BLACKLIST = ("Content-Length",) - -DEFAULT_REPRESENTATIONS = [("application/json", output_json)] - ## ... source file abbreviated to get to Headers examples ... + if self._has_fr_route(): + try: return self.handle_error(e) except Exception as f: return original_handler(f) return original_handler(e) def handle_error(self, e): - got_request_exception.send(current_app._get_current_object(), exception=e) - if ( not isinstance(e, HTTPException) and current_app.propagate_exceptions - and not isinstance(e, tuple(self.error_handlers.keys())) + and not isinstance(e, tuple(self._own_and_child_error_handlers.keys())) ): exc_type, exc_value, tb = sys.exc_info() @@ -227,6 +241,8 @@ DEFAULT_REPRESENTATIONS = [("application/json", output_json)] ) break else: + got_request_exception.send(current_app._get_current_object(), exception=e) + if isinstance(e, HTTPException): code = HTTPStatus(e.code) if include_message_in_response: @@ -240,8 +256,6 @@ DEFAULT_REPRESENTATIONS = [("application/json", output_json)] else: code = HTTPStatus.INTERNAL_SERVER_ERROR if include_message_in_response: - default_data = { - "message": code.phrase, ## ... source file continues with no further Headers examples... diff --git a/content/pages/examples/flask/flask-app-immutabledict.markdown b/content/pages/examples/flask/flask-app-immutabledict.markdown index 86bed27c1..d2a334fa8 100644 --- a/content/pages/examples/flask/flask-app-immutabledict.markdown +++ b/content/pages/examples/flask/flask-app-immutabledict.markdown @@ -40,8 +40,6 @@ The code is open sourced under the ```python # config.py -from __future__ import absolute_import, unicode_literals - import ast import codecs import os @@ -67,7 +65,6 @@ DEFAULTS = { 'ATTACHMENT_STORAGE': 'default', 'AUTH_PROVIDERS': {}, 'BASE_URL': None, - 'CACHE_BACKEND': 'files', 'CACHE_DIR': '/opt/indico/cache', 'CATEGORY_CLEANUP': {}, 'CELERY_BROKER': None, @@ -77,6 +74,7 @@ DEFAULTS = { 'CUSTOMIZATION_DEBUG': False, 'CUSTOMIZATION_DIR': None, 'CUSTOM_COUNTRIES': {}, + 'CUSTOM_LANGUAGES': {}, 'DB_LOG': False, 'DEBUG': False, @@ -86,12 +84,12 @@ DEFAULTS = { allowed |= set(INTERNAL_DEFAULTS) for key in set(data) - allowed: - warnings.warn('Ignoring unknown config key {}'.format(key)) - return {k: v for k, v in data.iteritems() if k in allowed} + warnings.warn(f'Ignoring unknown config key {key}') + return {k: v for k, v in data.items() if k in allowed} def load_config(only_defaults=False, override=None): - data = dict(DEFAULTS, **INTERNAL_DEFAULTS) + data = DEFAULTS | INTERNAL_DEFAULTS if not only_defaults: path = get_config_path() config = _sanitize_data(_parse_config(path)) @@ -112,7 +110,7 @@ def load_config(only_defaults=False, override=None): ~~ return ImmutableDict(data) -class IndicoConfig(object): +class IndicoConfig: __slots__ = ('_config', '_exc') diff --git a/content/pages/examples/flask/flask-blueprints-blueprint.markdown b/content/pages/examples/flask/flask-blueprints-blueprint.markdown index 62a85d083..16a0bb9c3 100644 --- a/content/pages/examples/flask/flask-blueprints-blueprint.markdown +++ b/content/pages/examples/flask/flask-blueprints-blueprint.markdown @@ -32,13 +32,13 @@ scenarios. CTFd is open sourced under the import base64 import requests -~~from flask import Blueprint +~~from flask import Blueprint, abort from flask import current_app as app from flask import redirect, render_template, request, session, url_for from itsdangerous.exc import BadSignature, BadTimeSignature, SignatureExpired from CTFd.cache import clear_team_session, clear_user_session -from CTFd.models import Teams, Users, db +from CTFd.models import Teams, UserFieldEntries, UserFields, Users, db from CTFd.utils import config, email, get_app_config, get_config from CTFd.utils import user as current_user from CTFd.utils import validators @@ -309,7 +309,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) backend, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling and many others. flask-base is provided as open source under the @@ -551,7 +551,7 @@ from werkzeug.urls import url_quote_plus from flask_debugtoolbar.compat import iteritems from flask_debugtoolbar.toolbar import DebugToolbar -from flask_debugtoolbar.utils import decode_text +from flask_debugtoolbar.utils import decode_text, gzip_compress, gzip_decompress try: from importlib.metadata import version @@ -649,7 +649,7 @@ def swagger_static(filename): ## Example 10 from Flask-WTF [Flask-WTF](https://github.com/lepture/flask-wtf) -([project documentation](https://flask-wtf.readthedocs.io/en/stable/) +([project documentation](https://flask-wtf.readthedocs.io/) and [PyPI page](https://pypi.org/project/Flask-WTF/)) provides a bridge between [Flask](/flask.html) and the the @@ -666,6 +666,7 @@ import hashlib import logging import os import warnings +from urllib.parse import urlparse from functools import wraps ~~from flask import Blueprint, current_app, g, request, session @@ -675,7 +676,7 @@ from werkzeug.security import safe_str_cmp from wtforms import ValidationError from wtforms.csrf.core import CSRF -from ._compat import FlaskWTFDeprecationWarning, string_types, urlparse +from ._compat import FlaskWTFDeprecationWarning __all__ = ('generate_csrf', 'validate_csrf', 'CSRFProtect') logger = logging.getLogger(__name__) @@ -714,7 +715,7 @@ def generate_csrf(secret_key=None, token_key=None): if not request.referrer: self._error_response('The referrer header is missing.') - good_referrer = 'https://{0}/'.format(request.host) + good_referrer = f'https://{request.host}/' if not same_origin(request.referrer, good_referrer): self._error_response('The referrer does not match the host.') @@ -727,7 +728,7 @@ def generate_csrf(secret_key=None, token_key=None): self._exempt_blueprints.add(view.name) return view - if isinstance(view, string_types): + if isinstance(view, str): view_location = view else: view_location = '.'.join((view.__module__, view.__name__)) diff --git a/content/pages/examples/flask/flask-cli-appgroup.markdown b/content/pages/examples/flask/flask-cli-appgroup.markdown index 828b6c8e2..0e496e286 100644 --- a/content/pages/examples/flask/flask-cli-appgroup.markdown +++ b/content/pages/examples/flask/flask-cli-appgroup.markdown @@ -35,8 +35,6 @@ The code is open sourced under the ```python # util.py -from __future__ import unicode_literals - import traceback from importlib import import_module @@ -50,14 +48,14 @@ from werkzeug.utils import cached_property def _create_app(info): from indico.web.flask.app import make_app - return make_app(set_path=True) + return make_app() class IndicoFlaskGroup(FlaskGroup): def __init__(self, **extra): - super(IndicoFlaskGroup, self).__init__(create_app=_create_app, add_default_commands=False, - add_version_option=False, set_debug_flag=False, **extra) + super().__init__(create_app=_create_app, add_default_commands=False, add_version_option=False, + set_debug_flag=False, **extra) self._indico_plugin_commands = None def _load_plugin_commands(self): @@ -65,7 +63,7 @@ class IndicoFlaskGroup(FlaskGroup): def _wrap_in_plugin_context(self, plugin, cmd): cmd.callback = wrap_in_plugin_context(plugin, cmd.callback) - for subcmd in getattr(cmd, 'commands', {}).viewvalues(): + for subcmd in getattr(cmd, 'commands', {}).values(): self._wrap_in_plugin_context(plugin, subcmd) def _get_indico_plugin_commands(self, ctx): @@ -77,12 +75,12 @@ class IndicoFlaskGroup(FlaskGroup): ctx.ensure_object(ScriptInfo).load_app() cmds = named_objects_from_signal(signals.plugin.cli.send(), plugin_attr='_indico_plugin') rv = {} - for name, cmd in cmds.viewitems(): + for name, cmd in cmds.items(): if cmd._indico_plugin: self._wrap_in_plugin_context(cmd._indico_plugin, cmd) rv[name] = cmd except Exception as exc: - if 'No indico config found' not in unicode(exc): + if 'No indico config found' not in str(exc): click.echo(click.style('Loading plugin commands failed:', fg='red', bold=True)) click.echo(click.style(traceback.format_exc(), fg='red')) rv = {} @@ -105,7 +103,7 @@ class LazyGroup(click.Group): def __init__(self, import_name, **kwargs): self._import_name = import_name - super(LazyGroup, self).__init__(**kwargs) + super().__init__(**kwargs) @cached_property def _impl(self): diff --git a/content/pages/examples/flask/flask-cli-dispatchingapp.markdown b/content/pages/examples/flask/flask-cli-dispatchingapp.markdown index 6301decdf..aa8140dd9 100644 --- a/content/pages/examples/flask/flask-cli-dispatchingapp.markdown +++ b/content/pages/examples/flask/flask-cli-dispatchingapp.markdown @@ -33,8 +33,6 @@ The code is open sourced under the ```python # devserver.py -from __future__ import print_function, unicode_literals - import os ~~from flask.cli import DispatchingApp @@ -92,7 +90,7 @@ def _make_wsgi_app(info, url, evalex_whitelist, proxy): return info.load_app() url_data = url_parse(url) -~~ app = DispatchingApp(_load_app) +~~ app = DispatchingApp(_load_app, use_eager_loading=False) app = DebuggedIndico(app, evalex_whitelist) app = _make_indico_dispatcher(app, url_data.path) if proxy: @@ -115,7 +113,7 @@ class DebuggedIndico(DebuggedApplication): def __init__(self, *args, **kwargs): self._evalex_whitelist = None self._request_ip = None - super(DebuggedIndico, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) diff --git a/content/pages/examples/flask/flask-cli-flaskgroup.markdown b/content/pages/examples/flask/flask-cli-flaskgroup.markdown index f42799cb7..1b39ccb93 100644 --- a/content/pages/examples/flask/flask-cli-flaskgroup.markdown +++ b/content/pages/examples/flask/flask-cli-flaskgroup.markdown @@ -43,8 +43,6 @@ import traceback from datetime import datetime import click -import click_log -from celery.bin.celery import CeleryCommand from flask import current_app ~~from flask.cli import FlaskGroup, ScriptInfo, with_appcontext from flask_alembic import alembic_click @@ -53,20 +51,29 @@ from sqlalchemy_utils.functions import database_exists from werkzeug.utils import import_string from flaskbb import create_app -from flaskbb.cli.utils import (EmailType, FlaskBBCLIError, get_version, - prompt_config_path, prompt_save_user, - write_config) +from flaskbb.cli.utils import ( + EmailType, + FlaskBBCLIError, + get_version, + prompt_config_path, + prompt_save_user, + write_config, +) from flaskbb.extensions import alembic, celery, db, whooshee -from flaskbb.utils.populate import (create_default_groups, - create_default_settings, create_latest_db, - create_test_data, create_welcome_forum, - insert_bulk_data, run_plugin_migrations, - update_settings_from_fixture) +from flaskbb.utils.populate import ( + create_default_groups, + create_default_settings, + create_latest_db, + create_test_data, + create_welcome_forum, + insert_bulk_data, + run_plugin_migrations, + update_settings_from_fixture, +) from flaskbb.utils.translations import compile_translations logger = logging.getLogger(__name__) -click_log.basic_config(logger) ~~class FlaskBBGroup(FlaskGroup): @@ -84,8 +91,7 @@ click_log.basic_config(logger) self._loaded_flaskbb_plugins = True except Exception: logger.error( - "Error while loading CLI Plugins", - exc_info=traceback.format_exc() + "Error while loading CLI Plugins", exc_info=traceback.format_exc() ) else: shell_context_processors = app.pluggy.hook.flaskbb_shell_context() @@ -94,6 +100,7 @@ click_log.basic_config(logger) def get_command(self, ctx, name): self._load_flaskbb_plugins(ctx) + return super(FlaskBBGroup, self).get_command(ctx, name) ## ... source file continues with no further FlaskGroup examples... @@ -115,8 +122,6 @@ The code is open sourced under the ```python # util.py -from __future__ import unicode_literals - import traceback from importlib import import_module @@ -130,14 +135,14 @@ from werkzeug.utils import cached_property def _create_app(info): from indico.web.flask.app import make_app - return make_app(set_path=True) + return make_app() ~~class IndicoFlaskGroup(FlaskGroup): def __init__(self, **extra): - super(IndicoFlaskGroup, self).__init__(create_app=_create_app, add_default_commands=False, - add_version_option=False, set_debug_flag=False, **extra) + super().__init__(create_app=_create_app, add_default_commands=False, add_version_option=False, + set_debug_flag=False, **extra) self._indico_plugin_commands = None def _load_plugin_commands(self): @@ -145,7 +150,7 @@ def _create_app(info): def _wrap_in_plugin_context(self, plugin, cmd): cmd.callback = wrap_in_plugin_context(plugin, cmd.callback) - for subcmd in getattr(cmd, 'commands', {}).viewvalues(): + for subcmd in getattr(cmd, 'commands', {}).values(): self._wrap_in_plugin_context(plugin, subcmd) def _get_indico_plugin_commands(self, ctx): @@ -157,7 +162,7 @@ def _create_app(info): ctx.ensure_object(ScriptInfo).load_app() cmds = named_objects_from_signal(signals.plugin.cli.send(), plugin_attr='_indico_plugin') rv = {} - for name, cmd in cmds.viewitems(): + for name, cmd in cmds.items(): ## ... source file continues with no further FlaskGroup examples... diff --git a/content/pages/examples/flask/flask-cli-pass-script-info.markdown b/content/pages/examples/flask/flask-cli-pass-script-info.markdown index c8516a952..2467707c5 100644 --- a/content/pages/examples/flask/flask-cli-pass-script-info.markdown +++ b/content/pages/examples/flask/flask-cli-pass-script-info.markdown @@ -32,15 +32,12 @@ The code is open sourced under the ```python # core.py -from __future__ import unicode_literals - import click ~~from flask.cli import AppGroup, pass_script_info from indico.cli.util import IndicoFlaskGroup, LazyGroup -click.disable_unicode_literals_warning = True __all__ = ('cli_command', 'cli_group') @@ -54,12 +51,13 @@ def _get_indico_version(ctx, param, value): if not value or ctx.resilient_parsing: return import indico - message = 'Indico v{}'.format(indico.__version__) + message = f'Indico v{indico.__version__}' click.echo(message, ctx.color) ctx.exit() @click.group(cls=IndicoFlaskGroup) +@click.option('--version', '-v', expose_value=False, callback=_get_indico_version, is_flag=True, is_eager=True, ## ... source file abbreviated to get to pass_script_info examples ... diff --git a/content/pages/examples/flask/flask-cli-scriptinfo.markdown b/content/pages/examples/flask/flask-cli-scriptinfo.markdown index 97aae3a5a..469bcd7b5 100644 --- a/content/pages/examples/flask/flask-cli-scriptinfo.markdown +++ b/content/pages/examples/flask/flask-cli-scriptinfo.markdown @@ -43,8 +43,6 @@ import traceback from datetime import datetime import click -import click_log -from celery.bin.celery import CeleryCommand from flask import current_app ~~from flask.cli import FlaskGroup, ScriptInfo, with_appcontext from flask_alembic import alembic_click @@ -53,20 +51,29 @@ from sqlalchemy_utils.functions import database_exists from werkzeug.utils import import_string from flaskbb import create_app -from flaskbb.cli.utils import (EmailType, FlaskBBCLIError, get_version, - prompt_config_path, prompt_save_user, - write_config) +from flaskbb.cli.utils import ( + EmailType, + FlaskBBCLIError, + get_version, + prompt_config_path, + prompt_save_user, + write_config, +) from flaskbb.extensions import alembic, celery, db, whooshee -from flaskbb.utils.populate import (create_default_groups, - create_default_settings, create_latest_db, - create_test_data, create_welcome_forum, - insert_bulk_data, run_plugin_migrations, - update_settings_from_fixture) +from flaskbb.utils.populate import ( + create_default_groups, + create_default_settings, + create_latest_db, + create_test_data, + create_welcome_forum, + insert_bulk_data, + run_plugin_migrations, + update_settings_from_fixture, +) from flaskbb.utils.translations import compile_translations logger = logging.getLogger(__name__) -click_log.basic_config(logger) class FlaskBBGroup(FlaskGroup): @@ -84,8 +91,7 @@ class FlaskBBGroup(FlaskGroup): self._loaded_flaskbb_plugins = True except Exception: logger.error( - "Error while loading CLI Plugins", - exc_info=traceback.format_exc() + "Error while loading CLI Plugins", exc_info=traceback.format_exc() ) else: shell_context_processors = app.pluggy.hook.flaskbb_shell_context() @@ -101,7 +107,12 @@ class FlaskBBGroup(FlaskGroup): return super(FlaskBBGroup, self).list_commands(ctx) -def make_app(script_info): +def make_app(): + ctx = click.get_current_context(silent=True) + script_info = None + if ctx is not None: + script_info = ctx.obj + config_file = getattr(script_info, "config_file", None) instance_path = getattr(script_info, "instance_path", None) return create_app(config_file, instance_path) @@ -115,28 +126,28 @@ def set_instance(ctx, param, value): ~~ ctx.ensure_object(ScriptInfo).instance_path = value -@click.group(cls=FlaskBBGroup, create_app=make_app, add_version_option=False, - invoke_without_command=True) -@click.option("--config", expose_value=False, callback=set_config, - required=False, is_flag=False, is_eager=True, metavar="CONFIG", - help="Specify the config to use either in dotted module " - "notation e.g. 'flaskbb.configs.default.DefaultConfig' " - "or by using a path like '/path/to/flaskbb.cfg'") -@click.option("--instance", expose_value=False, callback=set_instance, - required=False, is_flag=False, is_eager=True, metavar="PATH", - help="Specify the instance path to use. By default the folder " - "'instance' next to the package or module is assumed to " - "be the instance path.") -@click.option("--version", expose_value=False, callback=get_version, - is_flag=True, is_eager=True, help="Show the FlaskBB version.") -@click.pass_context -@click_log.simple_verbosity_option(logger) -def flaskbb(ctx): - if ctx.invoked_subcommand is None: - click.echo(ctx.get_help()) - - -flaskbb.add_command(alembic_click, "db") +@click.group( + cls=FlaskBBGroup, + create_app=make_app, + add_version_option=False, + invoke_without_command=True, +) +@click.option( + "--config", + expose_value=False, + callback=set_config, + required=False, + is_flag=False, + is_eager=True, + metavar="CONFIG", + help="Specify the config to use either in dotted module " + "notation e.g. 'flaskbb.configs.default.DefaultConfig' " + "or by using a path like '/path/to/flaskbb.cfg'", +) +@click.option( + "--instance", + expose_value=False, + callback=set_instance, ## ... source file continues with no further ScriptInfo examples... @@ -158,8 +169,6 @@ The code is open sourced under the ```python # util.py -from __future__ import unicode_literals - import traceback from importlib import import_module @@ -173,14 +182,14 @@ from werkzeug.utils import cached_property def _create_app(info): from indico.web.flask.app import make_app - return make_app(set_path=True) + return make_app() class IndicoFlaskGroup(FlaskGroup): def __init__(self, **extra): - super(IndicoFlaskGroup, self).__init__(create_app=_create_app, add_default_commands=False, - add_version_option=False, set_debug_flag=False, **extra) + super().__init__(create_app=_create_app, add_default_commands=False, add_version_option=False, + set_debug_flag=False, **extra) self._indico_plugin_commands = None def _load_plugin_commands(self): @@ -188,7 +197,7 @@ class IndicoFlaskGroup(FlaskGroup): def _wrap_in_plugin_context(self, plugin, cmd): cmd.callback = wrap_in_plugin_context(plugin, cmd.callback) - for subcmd in getattr(cmd, 'commands', {}).viewvalues(): + for subcmd in getattr(cmd, 'commands', {}).values(): self._wrap_in_plugin_context(plugin, subcmd) def _get_indico_plugin_commands(self, ctx): @@ -200,12 +209,12 @@ class IndicoFlaskGroup(FlaskGroup): ~~ ctx.ensure_object(ScriptInfo).load_app() cmds = named_objects_from_signal(signals.plugin.cli.send(), plugin_attr='_indico_plugin') rv = {} - for name, cmd in cmds.viewitems(): + for name, cmd in cmds.items(): if cmd._indico_plugin: self._wrap_in_plugin_context(cmd._indico_plugin, cmd) rv[name] = cmd except Exception as exc: - if 'No indico config found' not in unicode(exc): + if 'No indico config found' not in str(exc): click.echo(click.style('Loading plugin commands failed:', fg='red', bold=True)) click.echo(click.style(traceback.format_exc(), fg='red')) rv = {} diff --git a/content/pages/examples/flask/flask-cli-with-appcontext.markdown b/content/pages/examples/flask/flask-cli-with-appcontext.markdown index 70e615275..e2d1ab675 100644 --- a/content/pages/examples/flask/flask-cli-with-appcontext.markdown +++ b/content/pages/examples/flask/flask-cli-with-appcontext.markdown @@ -43,8 +43,6 @@ import traceback from datetime import datetime import click -import click_log -from celery.bin.celery import CeleryCommand from flask import current_app ~~from flask.cli import FlaskGroup, ScriptInfo, with_appcontext from flask_alembic import alembic_click @@ -53,35 +51,29 @@ from sqlalchemy_utils.functions import database_exists from werkzeug.utils import import_string from flaskbb import create_app -from flaskbb.cli.utils import (EmailType, FlaskBBCLIError, get_version, - prompt_config_path, prompt_save_user, - write_config) +from flaskbb.cli.utils import ( + EmailType, + FlaskBBCLIError, + get_version, + prompt_config_path, + prompt_save_user, + write_config, +) from flaskbb.extensions import alembic, celery, db, whooshee -from flaskbb.utils.populate import (create_default_groups, - create_default_settings, create_latest_db, - create_test_data, create_welcome_forum, - insert_bulk_data, run_plugin_migrations, - update_settings_from_fixture) -from flaskbb.utils.translations import compile_translations - - -logger = logging.getLogger(__name__) -click_log.basic_config(logger) - - -class FlaskBBGroup(FlaskGroup): - def __init__(self, *args, **kwargs): +from flaskbb.utils.populate import ( + create_default_groups, + create_default_settings, + create_latest_db, + create_test_data, + create_welcome_forum, + insert_bulk_data, + run_plugin_migrations, + update_settings_from_fixture, ## ... source file abbreviated to get to with_appcontext examples ... - "'instance' next to the package or module is assumed to " - "be the instance path.") -@click.option("--version", expose_value=False, callback=get_version, - is_flag=True, is_eager=True, help="Show the FlaskBB version.") -@click.pass_context -@click_log.simple_verbosity_option(logger) def flaskbb(ctx): if ctx.invoked_subcommand is None: click.echo(ctx.get_help()) @@ -91,23 +83,40 @@ flaskbb.add_command(alembic_click, "db") @flaskbb.command() -@click.option("--welcome", "-w", default=True, is_flag=True, - help="Disable the welcome forum.") -@click.option("--force", "-f", default=False, is_flag=True, - help="Doesn't ask for confirmation.") +@click.option( + "--welcome", "-w", default=True, is_flag=True, help="Disable the welcome forum." +) +@click.option( + "--force", "-f", default=False, is_flag=True, help="Doesn't ask for confirmation." +) @click.option("--username", "-u", help="The username of the user.") -@click.option("--email", "-e", type=EmailType(), - help="The email address of the user.") +@click.option("--email", "-e", type=EmailType(), help="The email address of the user.") @click.option("--password", "-p", help="The password of the user.") -@click.option("--no-plugins", "-n", default=False, is_flag=True, - help="Don't run the migrations for the default plugins.") +@click.option( + "--no-plugins", + "-n", + default=False, + is_flag=True, + help="Don't run the migrations for the default plugins.", +) ~~@with_appcontext def install(welcome, force, username, email, password, no_plugins): + if not current_app.config["CONFIG_PATH"]: + click.secho( + "[!] No 'flaskbb.cfg' config found. " + "You can generate a configuration file with 'flaskbb makeconfig'.", + fg="red", + ) + sys.exit(1) + click.secho("[+] Installing FlaskBB...", fg="cyan") if database_exists(db.engine.url): - if force or click.confirm(click.style( - "Existing database found. Do you want to delete the old one and " - "create a new one?", fg="magenta") + if force or click.confirm( + click.style( + "Existing database found. Do you want to delete the old one and " + "create a new one?", + fg="magenta", + ) ): db.drop_all() else: @@ -115,58 +124,46 @@ def install(welcome, force, username, email, password, no_plugins): create_latest_db() - click.secho("[+] Creating default settings...", fg="cyan") - create_default_groups() - create_default_settings() - - click.secho("[+] Creating admin user...", fg="cyan") - prompt_save_user(username, email, password, "admin") - - if welcome: - click.secho("[+] Creating welcome forum...", fg="cyan") - create_welcome_forum() - ## ... source file abbreviated to get to with_appcontext examples ... - - if fixture or all_latest: - try: - settings = import_string( - "flaskbb.fixtures.{}".format(fixture) - ) - settings = settings.fixture except ImportError: - raise FlaskBBCLIError("{} fixture is not available" - .format(fixture), fg="red") + raise FlaskBBCLIError( + "{} fixture is not available".format(fixture), fg="red" + ) click.secho("[+] Updating fixtures...", fg="cyan") count = update_settings_from_fixture( fixture=settings, overwrite_group=force, overwrite_setting=force ) - click.secho("[+] {settings} settings in {groups} setting groups " - "updated.".format(groups=len(count), settings=sum( - len(settings) for settings in count.values()) - ), fg="green") + click.secho( + "[+] {settings} settings in {groups} setting groups " + "updated.".format( + groups=len(count), + settings=sum(len(settings) for settings in count.values()), + ), + fg="green", + ) -@flaskbb.command("celery", add_help_option=False, - context_settings={"ignore_unknown_options": True, - "allow_extra_args": True}) +@flaskbb.command( + "celery", + add_help_option=False, + context_settings={"ignore_unknown_options": True, "allow_extra_args": True}, +) @click.pass_context ~~@with_appcontext def start_celery(ctx): - CeleryCommand(celery).execute_from_commandline( - ["flaskbb celery"] + ctx.args - ) + celery.start(ctx.args) @flaskbb.command("shell", short_help="Runs a shell in the app context.") ~~@with_appcontext def shell_command(): import code + banner = "Python %s on %s\nInstance Path: %s" % ( sys.version, sys.platform, @@ -183,25 +180,31 @@ def shell_command(): try: import IPython - IPython.embed(banner1=banner, user_ns=ctx) + from traitlets.config import get_config + + c = get_config() + c.InteractiveShellEmbed.colors = "Linux" + IPython.embed(config=c, banner1=banner, user_ns=ctx) except ImportError: code.interact(banner=banner, local=ctx) @flaskbb.command("urls", short_help="Show routes for the app.") -@click.option("--route", "-r", "order_by", flag_value="rule", default=True, - help="Order by route") -@click.option("--endpoint", "-e", "order_by", flag_value="endpoint", - help="Order by endpoint") -@click.option("--methods", "-m", "order_by", flag_value="methods", - help="Order by methods") +@click.option( + "--route", "-r", "order_by", flag_value="rule", default=True, help="Order by route" +) +@click.option( + "--endpoint", "-e", "order_by", flag_value="endpoint", help="Order by endpoint" +) +@click.option( + "--methods", "-m", "order_by", flag_value="methods", help="Order by methods" +) ~~@with_appcontext def list_urls(order_by): from flask import current_app rules = sorted( - current_app.url_map.iter_rules(), - key=lambda rule: getattr(rule, order_by) + current_app.url_map.iter_rules(), key=lambda rule: getattr(rule, order_by) ) max_rule_len = max(len(rule.rule) for rule in rules) @@ -215,11 +218,12 @@ def list_urls(order_by): column_header_len = max_rule_len + max_endpoint_len + max_method_len + 4 column_template = "{:<%s} {:<%s} {:<%s}" % ( - max_rule_len, max_endpoint_len, max_method_len + max_rule_len, + max_endpoint_len, + max_method_len, ) - click.secho(column_template.format("Route", "Endpoint", "Methods"), - fg="blue", bold=True) + click.secho( ## ... source file continues with no further with_appcontext examples... diff --git a/content/pages/examples/flask/flask-ctx-after-this-request.markdown b/content/pages/examples/flask/flask-ctx-after-this-request.markdown index 636546cb0..f18d7b9b7 100644 --- a/content/pages/examples/flask/flask-ctx-after-this-request.markdown +++ b/content/pages/examples/flask/flask-ctx-after-this-request.markdown @@ -40,33 +40,34 @@ The Flask-Security-Too project is provided as open source under the # unified_signin.py import time +import typing as t from flask import current_app as app -~~from flask import abort, after_this_request, request, session +~~from flask import after_this_request, request, session from flask_login import current_user from werkzeug.datastructures import MultiDict -from werkzeug.local import LocalProxy from wtforms import BooleanField, RadioField, StringField, SubmitField, validators from .confirmable import requires_confirmation from .decorators import anonymous_user_required, auth_required, unauth_csrf from .forms import Form, Required, get_form_field_label +from .proxies import _security, _datastore from .quart_compat import get_quart_status from .signals import us_profile_changed, us_security_token_sent -from .twofactor import is_tf_setup, tf_login +from .twofactor import ( + is_tf_setup, + tf_login, + tf_verify_validility_token, +) from .utils import ( _, SmsSenderFactory, base_render_json, check_and_get_token_status, - config_value, + config_value as cv, do_flash, find_user, get_identity_attributes, - get_post_login_redirect, - get_post_verify_redirect, - get_message, - get_url, ## ... source file abbreviated to get to after_this_request examples ... @@ -84,7 +85,7 @@ from .utils import ( token=self.passcode.data, totp_secret=self.totp_secret, user=self.user, - window=config_value("US_TOKEN_VALIDITY"), + window=cv("US_TOKEN_VALIDITY"), ): self.passcode.errors.append(get_message("INVALID_PASSWORD_CODE")[0]) return False @@ -97,7 +98,7 @@ def _send_code_helper(form): method = form.chosen_method.data totp_secrets = _datastore.us_get_totp_secrets(user) if method == "email" and method not in totp_secrets: -~~ after_this_request(_commit) +~~ after_this_request(view_commit) totp_secrets[method] = _security._totp_factory.generate_totp_secret() _datastore.us_put_totp_secrets(user, totp_secrets) @@ -127,32 +128,32 @@ def us_signin_send_code(): ## ... source file abbreviated to get to after_this_request examples ... - else: - return redirect(get_post_login_redirect()) - - form_class = _security.us_signin_form - - if request.is_json: - if request.content_length: - form = form_class(MultiDict(request.get_json()), meta=suppress_form_csrf()) - else: - form = form_class(formdata=None, meta=suppress_form_csrf()) - else: - form = form_class(meta=suppress_form_csrf()) form.submit.data = True if form.validate_on_submit(): + remember_me = form.remember.data if "remember" in form else None - if ( - config_value("TWO_FACTOR") - and form.authn_via in config_value("US_MFA_REQUIRED") - and (config_value("TWO_FACTOR_REQUIRED") or is_tf_setup(form.user)) - ): - return tf_login( - form.user, remember=remember_me, primary_authn_via=form.authn_via + if cv("TWO_FACTOR") and form.authn_via in cv("US_MFA_REQUIRED"): + if request.is_json and request.content_length: + tf_validity_token = request.get_json().get( # type: ignore + "tf_validity_token", None + ) + else: + tf_validity_token = request.cookies.get("tf_validity", default=None) + + tf_validity_token_is_valid = tf_verify_validility_token( + tf_validity_token, form.user.fs_uniquifier ) + if cv("TWO_FACTOR_REQUIRED") or is_tf_setup(form.user): + if cv("TWO_FACTOR_ALWAYS_VALIDATE") or (not tf_validity_token_is_valid): + + return tf_login( + form.user, + remember=remember_me, + primary_authn_via=form.authn_via, + ) -~~ after_this_request(_commit) +~~ after_this_request(view_commit) login_user(form.user, remember=remember_me, authn_via=[form.authn_via]) if _security._want_json(request): @@ -163,7 +164,7 @@ def us_signin_send_code(): code_methods = _compute_code_methods() if _security._want_json(request): payload = { - "available_methods": config_value("US_ENABLED_METHODS"), + "available_methods": cv("US_ENABLED_METHODS"), "code_methods": code_methods, "identity_attributes": get_identity_attributes(), } @@ -173,10 +174,10 @@ def us_signin_send_code(): return redirect(get_post_login_redirect()) form.passcode.data = None - return _security.render_template( - config_value("US_SIGNIN_TEMPLATE"), - us_signin_form=form, - available_methods=config_value("US_ENABLED_METHODS"), + + if form.requires_confirmation and cv("REQUIRES_CONFIRMATION_ERROR_VIEW"): + do_flash(*get_message("CONFIRMATION_REQUIRED")) + return redirect(get_url(cv("REQUIRES_CONFIRMATION_ERROR_VIEW"))) ## ... source file abbreviated to get to after_this_request examples ... @@ -185,7 +186,7 @@ def us_signin_send_code(): if _security.redirect_behavior == "spa": return redirect( get_url( - _security.login_error_view, + cv("LOGIN_ERROR_VIEW"), qparams=user.get_redirect_qparams({c: m}), ) ) @@ -193,24 +194,24 @@ def us_signin_send_code(): return redirect(url_for_security("us_signin")) if ( - config_value("TWO_FACTOR") - and "email" in config_value("US_MFA_REQUIRED") - and (config_value("TWO_FACTOR_REQUIRED") or is_tf_setup(user)) + cv("TWO_FACTOR") + and "email" in cv("US_MFA_REQUIRED") + and (cv("TWO_FACTOR_REQUIRED") or is_tf_setup(user)) ): if _security.redirect_behavior == "spa": return redirect( get_url( - _security.login_error_view, + cv("LOGIN_ERROR_VIEW"), qparams=user.get_redirect_qparams({"tf_required": 1}), ) ) return tf_login(user, primary_authn_via="email") login_user(user, authn_via=["email"]) -~~ after_this_request(_commit) +~~ after_this_request(view_commit) if _security.redirect_behavior == "spa": return redirect( - get_url(_security.post_login_view, qparams=user.get_redirect_qparams()) + get_url(cv("POST_LOGIN_VIEW"), qparams=user.get_redirect_qparams()) ) do_flash(*get_message("PASSWORDLESS_LOGIN_SUCCESSFUL")) @@ -218,10 +219,11 @@ def us_signin_send_code(): @auth_required( - within=lambda: config_value("FRESHNESS"), - grace=lambda: config_value("FRESHNESS_GRACE_PERIOD"), + lambda: cv("API_ENABLED_METHODS"), + within=lambda: cv("FRESHNESS"), + grace=lambda: cv("FRESHNESS_GRACE_PERIOD"), ) -def us_setup(): +def us_setup() -> "ResponseValue": form_class = _security.us_setup_form if request.is_json: @@ -233,7 +235,6 @@ def us_setup(): form = form_class(meta=suppress_form_csrf()) - ## ... source file abbreviated to get to after_this_request examples ... @@ -250,7 +251,7 @@ def us_setup(): if invalid: m, c = get_message("API_ERROR") if expired: - m, c = get_message("US_SETUP_EXPIRED", within=config_value("US_SETUP_WITHIN")) + m, c = get_message("US_SETUP_EXPIRED", within=cv("US_SETUP_WITHIN")) if invalid or expired: if _security._want_json(request): payload = json_error_response(errors=m) @@ -262,13 +263,13 @@ def us_setup(): form.user = current_user if form.validate_on_submit(): -~~ after_this_request(_commit) +~~ after_this_request(view_commit) method = state["chosen_method"] phone = state["phone_number"] if method == "sms" else None _datastore.us_set(current_user, method, state["totp_secret"], phone) us_profile_changed.send( - app._get_current_object(), user=current_user, method=method + app._get_current_object(), user=current_user, method=method # type: ignore ) if _security._want_json(request): return base_render_json( @@ -281,12 +282,12 @@ def us_setup(): else: do_flash(*get_message("US_SETUP_SUCCESSFUL")) return redirect( - get_url(_security.us_post_setup_view) - or get_url(_security.post_login_view) + get_url(cv("US_POST_SETUP_VIEW")) or get_url(cv("POST_LOGIN_VIEW")) ) if _security._want_json(request): return base_render_json(form, include_user=False) + m, c = get_message("INVALID_PASSWORD_CODE") ## ... source file continues with no further after_this_request examples... diff --git a/content/pages/examples/flask/flask-ctx-has-app-context.markdown b/content/pages/examples/flask/flask-ctx-has-app-context.markdown index 4742fb9f9..d4a03d102 100644 --- a/content/pages/examples/flask/flask-ctx-has-app-context.markdown +++ b/content/pages/examples/flask/flask-ctx-has-app-context.markdown @@ -143,21 +143,20 @@ The code is open sourced under the import ast import re -import textwrap -import traceback -import warnings +from collections import Counter from contextlib import contextmanager from babel import negotiate_locale from babel.core import LOCALE_ALIASES, Locale from babel.messages.pofile import read_po -from babel.support import NullTranslations, Translations +from babel.support import NullTranslations ~~from flask import current_app, g, has_app_context, has_request_context, request, session -from flask_babelex import Babel, Domain, get_domain +from flask_babel import Babel, Domain, get_domain from flask_pluginengine import current_plugin from speaklater import is_lazy_string, make_lazy_string from werkzeug.utils import cached_property +from indico.core.config import config from indico.util.caching import memoize_request @@ -182,22 +181,22 @@ def get_translation_domain(plugin_name=_use_context): return get_domain() -def gettext_unicode(*args, **kwargs): - from indico.util.string import inject_unicode_debug - func_name = kwargs.pop('func_name', 'ugettext') +def _indico_gettext(*args, **kwargs): + func_name = kwargs.pop('func_name', 'gettext') plugin_name = kwargs.pop('plugin_name', None) - force_unicode = kwargs.pop('force_unicode', False) - - if not isinstance(args[0], unicode): - args = [(text.decode('utf-8') if isinstance(text, str) else text) for text in args] - using_unicode = force_unicode - else: - using_unicode = True translations = get_translation_domain(plugin_name).get_translations() - res = getattr(translations, func_name)(*args, **kwargs) - res = inject_unicode_debug(res) - if not using_unicode: + return getattr(translations, func_name)(*args, **kwargs) + + +def lazy_gettext(string, plugin_name=None): + if is_lazy_string(string): + return string + return make_lazy_string(_indico_gettext, string, plugin_name=plugin_name) + + +def orig_string(lazy_string): + return lazy_string._args[0] if is_lazy_string(lazy_string) else lazy_string ## ... source file continues with no further has_app_context examples... diff --git a/content/pages/examples/flask/flask-ctx-has-request-context.markdown b/content/pages/examples/flask/flask-ctx-has-request-context.markdown index b639f67eb..9fe1163e0 100644 --- a/content/pages/examples/flask/flask-ctx-has-request-context.markdown +++ b/content/pages/examples/flask/flask-ctx-has-request-context.markdown @@ -179,7 +179,128 @@ def _get_item(model, view_arg, name): ``` -## Example 3 from indico +## Example 3 from Flask-SocketIO +[Flask-SocketIO](https://github.com/miguelgrinberg/Flask-SocketIO) +([PyPI package information](https://pypi.org/project/Flask-SocketIO/), +[official tutorial](https://blog.miguelgrinberg.com/post/easy-websockets-with-flask-and-gevent) +and +[project documentation](https://flask-socketio.readthedocs.io/en/latest/)) +is a code library by [Miguel Grinberg](https://blog.miguelgrinberg.com/index) +that provides Socket.IO integration for [Flask](/flask.html) applications. +This extension makes it easier to add bi-directional communications on the +web via the [WebSockets](/websockets.html) protocol. + +The Flask-SocketIO project is open source under the +[MIT license](https://github.com/miguelgrinberg/Flask-SocketIO/blob/master/LICENSE). + +[**Flask-SocketIO / flask_socketio / __init__.py**](https://github.com/miguelgrinberg/Flask-SocketIO/blob/master/./flask_socketio/__init__.py) + +```python +# __init__.py +from functools import wraps +import os +import sys + +gevent_socketio_found = True +try: + from socketio import socketio_manage # noqa: F401 +except ImportError: + gevent_socketio_found = False +if gevent_socketio_found: + print('The gevent-socketio package is incompatible with this version of ' + 'the Flask-SocketIO extension. Please uninstall it, and then ' + 'install the latest version of python-socketio in its place.') + sys.exit(1) + +import flask +~~from flask import _request_ctx_stack, has_request_context, json as flask_json +from flask.sessions import SessionMixin +import socketio +from socketio.exceptions import ConnectionRefusedError # noqa: F401 +from werkzeug.debug import DebuggedApplication +from werkzeug.serving import run_with_reloader + +from .namespace import Namespace +from .test_client import SocketIOTestClient + +__version__ = '5.0.2dev' + + +class _SocketIOMiddleware(socketio.WSGIApp): + def __init__(self, socketio_app, flask_app, socketio_path='socket.io'): + self.flask_app = flask_app + super(_SocketIOMiddleware, self).__init__(socketio_app, + flask_app.wsgi_app, + socketio_path=socketio_path) + + def __call__(self, environ, start_response): + environ = environ.copy() + environ['flask.app'] = self.flask_app + return super(_SocketIOMiddleware, self).__call__(environ, + start_response) + + +## ... source file abbreviated to get to has_request_context examples ... + + + else: + def set_handler(handler): + return self.on(handler.__name__, *args, **kwargs)(handler) + + return set_handler + + def on_namespace(self, namespace_handler): + if not isinstance(namespace_handler, Namespace): + raise ValueError('Not a namespace instance.') + namespace_handler._set_socketio(self) + if self.server: + self.server.register_namespace(namespace_handler) + else: + self.namespace_handlers.append(namespace_handler) + + def emit(self, event, *args, **kwargs): + namespace = kwargs.pop('namespace', '/') + to = kwargs.pop('to', kwargs.pop('room', None)) + include_self = kwargs.pop('include_self', True) + skip_sid = kwargs.pop('skip_sid', None) + if not include_self and not skip_sid: + skip_sid = flask.request.sid + callback = kwargs.pop('callback', None) + if callback: + sid = None +~~ if has_request_context(): + sid = getattr(flask.request, 'sid', None) + original_callback = callback + + def _callback_wrapper(*args): + return self._handle_event(original_callback, None, namespace, + sid, *args) + + if sid: + callback = _callback_wrapper + self.server.emit(event, *args, namespace=namespace, to=to, + skip_sid=skip_sid, callback=callback, **kwargs) + + def send(self, data, json=False, namespace=None, to=None, + callback=None, include_self=True, skip_sid=None, **kwargs): + skip_sid = flask.request.sid if not include_self else skip_sid + if json: + self.emit('json', data, namespace=namespace, to=to, + skip_sid=skip_sid, callback=callback, **kwargs) + else: + self.emit('message', data, namespace=namespace, to=to, + skip_sid=skip_sid, callback=callback, **kwargs) + + def close_room(self, room, namespace=None): + self.server.close_room(room, namespace) + + +## ... source file continues with no further has_request_context examples... + +``` + + +## Example 4 from indico [indico](https://github.com/indico/indico) ([project website](https://getindico.io/), [documentation](https://docs.getindico.io/en/stable/installation/) @@ -193,47 +314,36 @@ The code is open sourced under the ```python # logger.py -from __future__ import unicode_literals - import logging import logging.config import logging.handlers import os -import smtplib import warnings -from email.mime.text import MIMEText -from email.utils import formatdate from pprint import pformat import yaml -~~from flask import current_app, has_request_context, request, session +~~from flask import has_request_context, request, session from indico.core.config import config -from indico.util.i18n import set_best_lang -from indico.web.util import get_request_info +from indico.web.util import get_request_info, get_request_user -try: - from raven import setup_logging - from raven.contrib.celery import register_logger_signal, register_signal - from raven.contrib.flask import Sentry - from raven.handlers.logging import SentryHandler -except ImportError: - Sentry = object # so we can subclass - has_sentry = False -else: - has_sentry = True +class AddRequestIDFilter: + def filter(self, record): +~~ record.request_id = request.id if has_request_context() else '0' * 16 + return True -class AddRequestIDFilter(object): +class AddUserIDFilter: def filter(self, record): -~~ record.request_id = request.id if has_request_context() else '0' * 16 +~~ user = get_request_user()[0] if has_request_context() else None + record.user_id = str(session.user.id) if user else '-' return True class RequestInfoFormatter(logging.Formatter): def format(self, record): - rv = super(RequestInfoFormatter, self).format(record) + rv = super().format(record) info = get_request_info() if info: rv += '\n\n' + pformat(info) @@ -244,77 +354,13 @@ class FormattedSubjectSMTPHandler(logging.handlers.SMTPHandler): def getSubject(self, record): return self.subject % record.__dict__ - def emit(self, record): - try: - port = self.mailport - if not port: - port = smtplib.SMTP_PORT - smtp = smtplib.SMTP(self.mailhost, port, timeout=self._timeout) - msg = MIMEText(self.format(record), 'plain', 'utf-8') - msg['From'] = self.fromaddr - - -## ... source file abbreviated to get to has_request_context examples ... - - - if formatter.pop('append_request_info', False): - assert '()' not in formatter - formatter['()'] = RequestInfoFormatter - if config.DB_LOG: - data['loggers']['indico._db'] = {'level': 'DEBUG', 'propagate': False, 'handlers': ['_db']} - data['handlers']['_db'] = {'class': 'logging.handlers.SocketHandler', 'host': '127.0.0.1', 'port': 9020} - if config.CUSTOMIZATION_DEBUG and config.CUSTOMIZATION_DIR: - data['loggers'].setdefault('indico.customization', {})['level'] = 'DEBUG' - logging.config.dictConfig(data) - if config.SENTRY_DSN: - if not has_sentry: - raise Exception('`raven` must be installed to use sentry logging') - init_sentry(app) - - @classmethod - def get(cls, name=None): - if name is None: - name = 'indico' - elif name != 'indico' and not name.startswith('indico.'): - name = 'indico.' + name - return logging.getLogger(name) - - -class IndicoSentry(Sentry): - def get_user_info(self, request): -~~ if not has_request_context() or not session.user: - return None - return {'id': session.user.id, - 'email': session.user.email, - 'name': session.user.full_name} - - def before_request(self, *args, **kwargs): - super(IndicoSentry, self).before_request() -~~ if not has_request_context(): - return - self.client.extra_context({'Endpoint': str(request.url_rule.endpoint) if request.url_rule else None, - 'Request ID': request.id}) - self.client.tags_context({'locale': set_best_lang()}) - - -def init_sentry(app): - sentry = IndicoSentry(wrap_wsgi=False, register_signal=True, logging=False) - sentry.init_app(app) - handler = SentryHandler(sentry.client, level=getattr(logging, config.SENTRY_LOGGING_LEVEL)) - handler.addFilter(BlacklistFilter({'indico.flask', 'celery.redirected'})) - setup_logging(handler) - register_logger_signal(sentry.client) - register_signal(sentry.client) - - -def sentry_log_exception(): - try: - sentry = current_app.extensions['sentry'] - except KeyError: - return - sentry.captureException() +class BlacklistFilter(logging.Filter): + def __init__(self, names): + self.filters = [logging.Filter(name) for name in names] + def filter(self, record): + return not any(x.filter(record) for x in self.filters) ## ... source file continues with no further has_request_context examples... diff --git a/content/pages/examples/flask/flask-example-projects-code.markdown b/content/pages/examples/flask/flask-example-projects-code.markdown index 438fc76c6..e8a469a5d 100644 --- a/content/pages/examples/flask/flask-example-projects-code.markdown +++ b/content/pages/examples/flask/flask-example-projects-code.markdown @@ -99,7 +99,7 @@ is configured in JSON. The code is provided as open source under the is an example application that ties together the [intTellInput.js](https://github.com/jackocnr/intl-tel-input) JavaScript plugin with the -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) form-handling +[Flask-WTF](https://flask-wtf.readthedocs.io/) form-handling library. flask-phone-input is provided as open source under the [MIT license](https://github.com/miguelgrinberg/flask-phone-input/blob/1a1c227c044474ce0fe133493d7f8b0fb8312409/LICENSE). @@ -158,6 +158,14 @@ source under the [GNU General Public License](https://github.com/danielhomola/science_flask/blob/master/LICENSE). +### ShortMe-URL-Shortener +[ShortMe](https://github.com/AcrobaticPanicc/ShortMe-URL-Shortener) +is a [Flask](/flask.html) app that creates a shortened URL +that redirects to another, typically much longer, URL. The +project is provided as open source under the +[MIT license](https://github.com/AcrobaticPanicc/ShortMe-URL-Shortener/blob/main/LICENSE). + + ### tedivm's flask starter app [tedivm's flask starter app](https://github.com/tedivm/tedivms-flask) is a base of [Flask](/flask.html) code and related projects such as diff --git a/content/pages/examples/flask/flask-extensions-plug-ins.markdown b/content/pages/examples/flask/flask-extensions-plug-ins.markdown index 67c8b60c2..de0d41668 100644 --- a/content/pages/examples/flask/flask-extensions-plug-ins.markdown +++ b/content/pages/examples/flask/flask-extensions-plug-ins.markdown @@ -54,6 +54,17 @@ is provided as open source under the [Apache 2.0 license](https://github.com/johnwheeler/flask-ask/blob/master/LICENSE.txt). +### Flask-Authorize +[Flask-Authorize](https://github.com/bprinty/Flask-Authorize) +([documentation](https://flask-authorize.readthedocs.io/en/latest/) +and +[PyPI package](https://pypi.org/project/Flask-Authorize/)) +is a [Flask](/flask.html) extension to make it easier to implement +Access Control Lists (ACLs) and Role-Based Access Control (RBAC) into +web applications. The project is open sourced under the +[MIT license](https://github.com/bprinty/Flask-Authorize/blob/master/LICENSE). + + ### flask-base [flask-base](https://github.com/hack4impact/flask-base) ([project documentation](http://hack4impact.github.io/flask-base/)) @@ -114,6 +125,15 @@ open sourced under the [MIT license](https://github.com/maxcountryman/flask-login/blob/master/LICENSE). +### Flask-Meld +[Flask-Meld](https://github.com/mikeabrahamsen/Flask-Meld) +([PyPI package information](https://pypi.org/project/Flask-Meld/)) +allows you to write your front end web code in your back end +Python code. It does this by adding a `{% meld_scripts %}` tag to +the Flask template engine and then inserting components written +in Python scripts created by a developer. + + ### flask-praetorian [flask-praetorian](https://github.com/dusktreader/flask-praetorian) ([project documentation](https://flask-praetorian.readthedocs.io/en/latest/) @@ -199,7 +219,7 @@ building on, and the source code is open source under the ### Flask-WTF [Flask-WTF](https://github.com/lepture/flask-wtf) -([project documentation](https://flask-wtf.readthedocs.io/en/stable/) +([project documentation](https://flask-wtf.readthedocs.io/) and [PyPI page](https://pypi.org/project/Flask-WTF/)) provides a bridge between [Flask](/flask.html) and the the diff --git a/content/pages/examples/flask/flask-globals-current-app.markdown b/content/pages/examples/flask/flask-globals-current-app.markdown index 6e9dd327b..450633b87 100644 --- a/content/pages/examples/flask/flask-globals-current-app.markdown +++ b/content/pages/examples/flask/flask-globals-current-app.markdown @@ -103,130 +103,327 @@ forms, and internationalization support. Flask App Builder is provided under the [BSD 3-Clause "New" or "Revised" license](https://github.com/dpgaspar/Flask-AppBuilder/blob/master/LICENSE). -[**Flask AppBuilder / flask_appbuilder / menu.py**](https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/./menu.py) +[**Flask AppBuilder / flask_appbuilder / validators.py**](https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/./validators.py) ```python -# menu.py -from typing import List - -~~from flask import current_app, url_for -from flask_babel import gettext as __ - -from .api import BaseApi, expose -from .basemanager import BaseManager -from .security.decorators import permission_name, protect - - -class MenuItem(object): - def __init__(self, name, href="", icon="", label="", childs=None, baseview=None): - self.name = name - self.href = href - self.icon = icon - self.label = label - self.childs = childs or [] - self.baseview = baseview - - def get_url(self): - if not self.href: - if not self.baseview: - return "" +# validators.py +import re +from typing import Optional + +~~from flask import current_app +from flask_appbuilder.exceptions import PasswordComplexityValidationError +from flask_appbuilder.models.base import BaseInterface +from flask_babel import gettext +from wtforms import Field, Form, ValidationError + +password_complexity_regex = re.compile( + r"""( + ^(?=.*[A-Z].*[A-Z]) # at least two capital letters + (?=.*[^0-9a-zA-Z]) # at least one of these special characters + (?=.*[0-9].*[0-9]) # at least two numeric digits + (?=.*[a-z].*[a-z].*[a-z]) # at least three lower case letters + .{10,} # at least 10 total characters + $ + )""", + re.VERBOSE, +) + + +class Unique: + + field_flags = ("unique",) + + def __init__( + self, datamodel: BaseInterface, col_name: str, message: Optional[str] = None + ) -> None: + self.datamodel = datamodel + self.col_name = col_name + self.message = message + + def __call__(self, form: Form, field: Field) -> None: + filters = self.datamodel.get_filters().add_filter( + self.col_name, self.datamodel.FilterEqual, field.data + ) + count, obj = self.datamodel.query(filters) + if count > 0: + if not hasattr(form, "_id") or form._id != self.datamodel.get_keys(obj)[0]: + if self.message is None: + self.message = field.gettext(u"Already exists.") + raise ValidationError(self.message) + + +class PasswordComplexityValidator: + + def __call__(self, form: Form, field: Field) -> None: +~~ if current_app.config.get("FAB_PASSWORD_COMPLEXITY_ENABLED", False): +~~ password_complexity_validator = current_app.config.get( + "FAB_PASSWORD_COMPLEXITY_VALIDATOR", None + ) + if password_complexity_validator is not None: + try: + password_complexity_validator(field.data) + except PasswordComplexityValidationError as exc: + raise ValidationError(str(exc)) else: - return url_for(f"{self.baseview.endpoint}.{self.baseview.default_view}") - else: - try: + try: + default_password_complexity(field.data) + except PasswordComplexityValidationError as exc: + raise ValidationError(str(exc)) + + +def default_password_complexity(password: str) -> None: + match = re.search(password_complexity_regex, password) + if not match: + raise PasswordComplexityValidationError( + gettext( + "Must have at least two capital letters," + " one special character, two digits, three lower case letters and" + " a minimal length of 10." + ) + ) -## ... source file abbreviated to get to current_app examples ... +## ... source file continues with no further current_app examples... +``` + + +## Example 3 from Flask-Authorize +[Flask-Authorize](https://github.com/bprinty/Flask-Authorize) +([documentation](https://flask-authorize.readthedocs.io/en/latest/) +and +[PyPI package](https://pypi.org/project/Flask-Authorize/)) +is a [Flask](/flask.html) extension to make it easier to implement +Access Control Lists (ACLs) and Role-Based Access Control (RBAC) into +web applications. The project is open sourced under the +[MIT license](https://github.com/bprinty/Flask-Authorize/blob/master/LICENSE). + +[**Flask-Authorize / flask_authorize / mixins.py**](https://github.com/bprinty/Flask-Authorize/blob/master/flask_authorize/./mixins.py) + +```python +# mixins.py - self.menu = [] - if reverse: - extra_classes = extra_classes + "navbar-inverse" - self.extra_classes = extra_classes + +import six +import re +import json +~~from flask import current_app +from werkzeug.exceptions import Unauthorized +from sqlalchemy import Column, ForeignKey +from sqlalchemy.types import Integer, Text +from sqlalchemy.orm import relationship +from sqlalchemy.sql import operators +from sqlalchemy.ext.declarative import declared_attr +from sqlalchemy import TypeDecorator, inspect, and_, or_ + + +class JSON(TypeDecorator): + impl = Text @property - def reverse(self): - return "navbar-inverse" in self.extra_classes - - def get_list(self): - return self.menu - - def get_flat_name_list(self, menu: "Menu" = None, result: List = None) -> List: - menu = menu or self.menu - result = result or [] - for item in menu: - result.append(item.name) - if item.childs: - result.extend(self.get_flat_name_list(menu=item.childs, result=result)) - return result + def python_type(self): + return object - def get_data(self, menu=None): - menu = menu or self.menu - ret_list = [] + def process_bind_param(self, value, dialect): + return json.dumps(value) -~~ allowed_menus = current_app.appbuilder.sm.get_user_menu_access( - self.get_flat_name_list() - ) + def process_result_value(self, value, dialect): + try: + return json.loads(value) + except (ValueError, TypeError): + return None - for i, item in enumerate(menu): - if item.name == "-" and not i == len(menu) - 1: - ret_list.append("-") - elif item.name not in allowed_menus: + +## ... source file abbreviated to get to current_app examples ... + + + operators.contains_op): + return Text() + else: + return self + + def process_bind_param(self, value, dialect): + if not value: + return None + return '|'.join(value) + + def process_result_value(self, value, dialect): + try: + if not value: + return [] + return value.split('|') + except (ValueError, TypeError): + return None + + +MODELS = dict() + + +def gather_models(): + global MODELS + +~~ from flask import current_app +~~ if 'sqlalchemy' not in current_app.extensions: + return +~~ check = current_app.config['AUTHORIZE_IGNORE_PROPERTY'] + +~~ db = current_app.extensions['sqlalchemy'].db + for cls in db.Model._decl_class_registry.values(): + if isinstance(cls, type) and issubclass(cls, db.Model): + if hasattr(cls, check) and not getattr(cls, check): continue - elif item.childs: - ret_list.append( - { - "name": item.name, - "icon": item.icon, - "label": __(str(item.label)), - "childs": self.get_data(menu=item.childs), - } - ) - else: - ret_list.append( - { - "name": item.name, - "icon": item.icon, - "label": __(str(item.label)), - "url": item.get_url(), + MODELS[table_key(cls)] = cls + return + + +def table_key(cls): +~~ if current_app.config['AUTHORIZE_MODEL_PARSER'] == 'class': + return cls.__name__ + +~~ elif current_app.config['AUTHORIZE_MODEL_PARSER'] == 'lower': + return cls.__name__.lower() + +~~ elif current_app.config['AUTHORIZE_MODEL_PARSER'] == 'snake': + words = re.findall(r'([A-Z][0-9a-z]+)', cls.__name__) + if len(words) > 1: + return '_'.join(map(lambda x: x.lower(), words)) + +~~ elif current_app.config['AUTHORIZE_MODEL_PARSER'] == 'table': + mapper = inspect(cls) + return mapper.tables[0].name + + +def default_permissions_factory(name): + def _(cls=None): + perms = default_permissions(cls) + return perms.get(name, []) + return _ + + +def default_permissions(cls=None): + if cls is None or cls.__permissions__ is None: +~~ return current_app.config['AUTHORIZE_DEFAULT_PERMISSIONS'] + elif isinstance(cls.__permissions__, int): + return parse_permission_set(cls.__permissions__) + elif isinstance(cls.__permissions__, dict): + return cls.__permissions__ + + +def default_allowances(cls=None): + global MODELS + if not MODELS: + gather_models() + + default = { +~~ key: current_app.config['AUTHORIZE_DEFAULT_ALLOWANCES'] + for key in MODELS + } + + if cls is None: + return default + + if isinstance(cls.__allowances__, dict): + return cls.__allowances__ + + return default + + +def default_restrictions(cls=None): + global MODELS + if not MODELS: + gather_models() + + default = { +~~ key: current_app.config['AUTHORIZE_DEFAULT_RESTRICTIONS'] + for key in MODELS + } + + if cls is None: + return default + + if cls.__restrictions__ == '*' or cls.__restrictions__ is True: + return { +~~ key: current_app.config['AUTHORIZE_DEFAULT_ACTIONS'] + for key in MODELS + } + + if isinstance(cls.__restrictions__, dict): + default.update(cls.__restrictions__) + return default + + +def permission_list(number): + if isinstance(number, six.string_types) and len(number) == 1: + number = int(number) + if not isinstance(number, int): + return number + + ret = [] + for mask, name in zip([1, 2, 4], ['delete', 'read', 'update']): + if number & mask: + ret.append(name) + return ret + + +def parse_permission_set(number): + if isinstance(number, six.string_types) and len(number) == 3: + number = int(number) ## ... source file abbreviated to get to current_app examples ... - category=category, icon=category_icon, label=category_label - ) - new_menu_item = MenuItem( - name=name, href=href, icon=icon, label=label, baseview=baseview - ) - self.find(category).childs.append(new_menu_item) + cls.group_id.in_([x.id for x in current_user.groups]), + cls.group_permissions.contains(check) + )) + return or_(*clauses) - def add_separator(self, category=""): - menu_item = self.find(category) - if menu_item: - menu_item.childs.append(MenuItem("-")) - else: - raise Exception( - "Menu separator does not have correct category {}".format(category) - ) + @property + def permissions(self): + result = {} + for name in ['owner', 'group', 'other']: + prop = name + '_permissions' + if hasattr(self, prop): + result[name] = getattr(self, prop) + return result + + @permissions.setter + def permissions(self, value): + for name in ['owner', 'group', 'other']: + if name not in value: + continue + prop = name + '_permissions' + if hasattr(self, prop): + setattr(self, prop, value[name]) + return + def set_permissions(self, *args, **kwargs): +~~ if 'authorize' in current_app.extensions: +~~ authorize = current_app.extensions['authorize'] + if not authorize.update(self): + raise Unauthorized -class MenuApi(BaseApi): - resource_name = "menu" - openapi_spec_tag = "Menu" + if len(args): + perms = parse_permission_set(args[0]) + kwargs.update(perms) - @expose("/", methods=["GET"]) - @protect(allow_browser_login=True) - @permission_name("get") - def get_menu_data(self): -~~ return self.response(200, result=current_app.appbuilder.menu.get_data()) + permissions = self.permissions.copy() + permissions.update(kwargs) + self.permissions = permissions + return self -class MenuApiManager(BaseManager): - def register_views(self): - if self.appbuilder.app.config.get("FAB_ADD_MENU_API", True): - self.appbuilder.add_api(MenuApi) +class OwnerMixin(object): + __user_model__ = 'User' + @classmethod + def get_user_default(cls): + from .plugin import CURRENT_USER + return CURRENT_USER().id + + @classmethod + def get_user_tablename(cls): + if isinstance(cls.__user_model__, str): ## ... source file continues with no further current_app examples... @@ -234,7 +431,7 @@ class MenuApiManager(BaseManager): ``` -## Example 3 from FlaskBB +## Example 4 from FlaskBB [FlaskBB](https://github.com/flaskbb/flaskbb) ([project website](https://flaskbb.org/)) is a [Flask](/flask.html)-based forum web application. The web app allows users to chat in an open @@ -377,7 +574,7 @@ class ForgotPassword(MethodView): ``` -## Example 4 from flask-base +## Example 5 from flask-base [flask-base](https://github.com/hack4impact/flask-base) ([project documentation](http://hack4impact.github.io/flask-base/)) provides boilerplate code for new [Flask](/flask.html) web apps. @@ -387,7 +584,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) backend, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling and many others. flask-base is provided as open source under the @@ -484,7 +681,7 @@ class User(UserMixin, db.Model): ``` -## Example 5 from flask-bookshelf +## Example 6 from flask-bookshelf [flask-bookshelf](https://github.com/damyanbogoev/flask-bookshelf) is the example [Flask](/flask.html) application that developers create when going through @@ -589,7 +786,7 @@ app.register_blueprint(admin, url_prefix="//admin") ``` -## Example 6 from Flask-Bootstrap +## Example 7 from Flask-Bootstrap [flask-bootstrap](https://github.com/mbr/flask-bootstrap) ([PyPI package information](https://pypi.org/project/Flask-Bootstrap/)) makes it easier to use the [Bootstrap CSS framework](/bootstrap-css.html) @@ -713,7 +910,7 @@ class Bootstrap(object): ``` -## Example 7 from flask-debugtoolbar +## Example 8 from flask-debugtoolbar [Flask Debug-toolbar](https://github.com/flask-debugtoolbar/flask-debugtoolbar) ([documentation](https://flask-debugtoolbar.readthedocs.io/en/latest/) and @@ -739,7 +936,7 @@ from werkzeug.urls import url_quote_plus from flask_debugtoolbar.compat import iteritems from flask_debugtoolbar.toolbar import DebugToolbar -from flask_debugtoolbar.utils import decode_text +from flask_debugtoolbar.utils import decode_text, gzip_compress, gzip_decompress try: from importlib.metadata import version @@ -848,12 +1045,12 @@ def replace_insensitive(string, target, replacement): response.headers['content-type'].startswith('text/html')): return response - response_html = response.data.decode(response.charset) + if 'gzip' in response.headers.get('Content-Encoding', ''): + response_html = gzip_decompress(response.data).decode(response.charset) + else: + response_html = response.data.decode(response.charset) no_case = response_html.lower() - body_end = no_case.rfind('') - - if body_end >= 0: ## ... source file continues with no further current_app examples... @@ -861,7 +1058,7 @@ def replace_insensitive(string, target, replacement): ``` -## Example 8 from flask_jsondash +## Example 9 from flask_jsondash [Flask JSONDash](https://github.com/christabor/flask_jsondash) is a configurable web application built in Flask that creates charts and dashboards from arbitrary API endpoints. Everything for the web app @@ -954,7 +1151,7 @@ def local_static(chart_config, static_config): ``` -## Example 9 from flask-login +## Example 10 from flask-login [Flask-Login](https://github.com/maxcountryman/flask-login) ([project documentation](https://flask-login.readthedocs.io/en/latest/) and [PyPI package](https://pypi.org/project/Flask-Login/)) @@ -1201,7 +1398,88 @@ def _secret_key(key=None): ``` -## Example 10 from flask-restx +## Example 11 from Flask-Meld +[Flask-Meld](https://github.com/mikeabrahamsen/Flask-Meld) +([PyPI package information](https://pypi.org/project/Flask-Meld/)) +allows you to write your front end web code in your back end +Python code. It does this by adding a `{% meld_scripts %}` tag to +the Flask template engine and then inserting components written +in Python scripts created by a developer. + +[**Flask-Meld / flask_meld / message.py**](https://github.com/mikeabrahamsen/Flask-Meld/blob/main/flask_meld/./message.py) + +```python +# message.py +import ast +from werkzeug.wrappers.response import Response +import functools + +from .component import get_component_class +~~from flask import jsonify, current_app +import orjson + + +def process_message(message): + meld_id = message["id"] + component_name = message["componentName"] + action_queue = message["actionQueue"] + + data = message["data"] + Component = get_component_class(component_name) + component = Component(meld_id, **data) + return_data = None + + for action in action_queue: + payload = action.get("payload", None) + if "syncInput" in action["type"]: + if hasattr(component, payload["name"]): + setattr(component, payload["name"], payload["value"]) + if component._form: + field_name = payload.get("name") + if field_name in component._form._fields: + field = getattr(component._form, field_name) + component._set_field_data(field_name, payload["value"]) + component.updated(field) + + +## ... source file abbreviated to get to current_app examples ... + + + + if "(" in call_method_name and call_method_name.endswith(")"): + param_idx = call_method_name.index("(") + params_str = call_method_name[param_idx:] + + method_name = call_method_name.replace(params_str, "") + + params_str = params_str[1:-1] + if params_str != "": + try: + params = ast.literal_eval("[" + params_str + "]") + except (ValueError, SyntaxError): + params = list(map(str.strip, params_str.split(","))) + + return method_name, params + + +def listen(*event_names: str): + def dec(func): + func._meld_event_names = event_names + return func + return dec + + +def emit(event_name: str, **kwargs): +~~ current_app.socketio.emit("meld-event", {"event": event_name, "message": kwargs}) + + + +## ... source file continues with no further current_app examples... + +``` + + +## Example 12 from flask-restx [Flask RESTX](https://github.com/python-restx/flask-restx) is an extension that makes it easier to build [RESTful APIs](/application-programming-interfaces.html) into @@ -1234,7 +1512,7 @@ from six import string_types, itervalues, iteritems, iterkeys from werkzeug.routing import parse_rule from . import fields -from .model import Model, ModelBase +from .model import Model, ModelBase, OrderedModel from .reqparse import RequestParser from .utils import merge, not_none, not_none_sorted from ._http import HTTPStatus @@ -1458,7 +1736,7 @@ def _clean_header(header): ``` -## Example 11 from flask-sqlalchemy +## Example 13 from flask-sqlalchemy [flask-sqlalchemy](https://github.com/pallets/flask-sqlalchemy) ([project documentation](https://flask-sqlalchemy.palletsprojects.com/en/2.x/) and @@ -1495,24 +1773,24 @@ from sqlalchemy import event from sqlalchemy import inspect from sqlalchemy import orm from sqlalchemy.engine.url import make_url -from sqlalchemy.ext.declarative import declarative_base -from sqlalchemy.ext.declarative import DeclarativeMeta from sqlalchemy.orm.exc import UnmappedClassError from sqlalchemy.orm.session import Session as SessionBase from .model import DefaultMeta from .model import Model -__version__ = "3.0.0.dev" - -_signals = Namespace() -models_committed = _signals.signal("models-committed") -before_models_committed = _signals.signal("before-models-committed") +try: + from sqlalchemy.orm import declarative_base + from sqlalchemy.orm import DeclarativeMeta +except ImportError: + from sqlalchemy.ext.declarative import declarative_base + from sqlalchemy.ext.declarative import DeclarativeMeta +try: + from greenlet import getcurrent as _ident_func +except ImportError: + from threading import get_ident as _ident_func -def _make_table(db): - def _make_table(*args, **kwargs): - if len(args) > 1 and isinstance(args[1], db.Column): ## ... source file abbreviated to get to current_app examples ... @@ -1551,7 +1829,7 @@ def _make_table(db): raise RuntimeError( "No application found. Either work inside a view function or push" " an application context. See" - " http://flask-sqlalchemy.pocoo.org/contexts/." + " https://flask-sqlalchemy.palletsprojects.com/contexts/." ) def get_tables_for_bind(self, bind=None): @@ -1575,9 +1853,9 @@ def _make_table(db): ``` -## Example 12 from Flask-WTF +## Example 14 from Flask-WTF [Flask-WTF](https://github.com/lepture/flask-wtf) -([project documentation](https://flask-wtf.readthedocs.io/en/stable/) +([project documentation](https://flask-wtf.readthedocs.io/) and [PyPI page](https://pypi.org/project/Flask-WTF/)) provides a bridge between [Flask](/flask.html) and the the @@ -1594,6 +1872,7 @@ import hashlib import logging import os import warnings +from urllib.parse import urlparse from functools import wraps ~~from flask import Blueprint, current_app, g, request, session @@ -1603,7 +1882,7 @@ from werkzeug.security import safe_str_cmp from wtforms import ValidationError from wtforms.csrf.core import CSRF -from ._compat import FlaskWTFDeprecationWarning, string_types, urlparse +from ._compat import FlaskWTFDeprecationWarning __all__ = ('generate_csrf', 'validate_csrf', 'CSRFProtect') logger = logging.getLogger(__name__) @@ -1611,7 +1890,7 @@ logger = logging.getLogger(__name__) def generate_csrf(secret_key=None, token_key=None): - secret_key = _get_config( +~~ secret_key = _get_config( ~~ secret_key, 'WTF_CSRF_SECRET_KEY', current_app.secret_key, message='A secret key is required to use CSRF.' ) @@ -1639,7 +1918,7 @@ def generate_csrf(secret_key=None, token_key=None): def validate_csrf(data, secret_key=None, time_limit=None, token_key=None): - secret_key = _get_config( +~~ secret_key = _get_config( ~~ secret_key, 'WTF_CSRF_SECRET_KEY', current_app.secret_key, message='A secret key is required to use CSRF.' ) @@ -1687,7 +1966,7 @@ def _get_config( class _FlaskFormCSRF(CSRF): def setup_form(self, form): self.meta = form.meta - return super(_FlaskFormCSRF, self).setup_form(form) + return super().setup_form(form) def generate_csrf_token(self, csrf_token_field): return generate_csrf( @@ -1723,7 +2002,7 @@ class _FlaskFormCSRF(CSRF): return view = app.view_functions.get(request.endpoint) - dest = '{0}.{1}'.format(view.__module__, view.__name__) + dest = f'{view.__module__}.{view.__name__}' if dest in self._exempt_views: return @@ -1766,7 +2045,7 @@ class _FlaskFormCSRF(CSRF): if not request.referrer: self._error_response('The referrer header is missing.') - good_referrer = 'https://{0}/'.format(request.host) + good_referrer = f'https://{request.host}/' if not same_origin(request.referrer, good_referrer): self._error_response('The referrer does not match the host.') @@ -1779,7 +2058,7 @@ class _FlaskFormCSRF(CSRF): self._exempt_blueprints.add(view.name) return view - if isinstance(view, string_types): + if isinstance(view, str): view_location = view else: view_location = '.'.join((view.__module__, view.__name__)) @@ -1814,7 +2093,7 @@ class CsrfProtect(CSRFProtect): '"flask_wtf.CsrfProtect" has been renamed to "CSRFProtect" ' 'and will be removed in 1.0.' ), stacklevel=2) - super(CsrfProtect, self).__init__(app=app) + super().__init__(app=app) class CSRFError(BadRequest): @@ -1832,7 +2111,7 @@ def same_origin(current_uri, compare_uri): ``` -## Example 13 from Flask-Security-Too +## Example 15 from Flask-Security-Too [Flask-Security-Too](https://github.com/Flask-Middleware/flask-security/) ([PyPi page](https://pypi.org/project/Flask-Security-Too/) and [project documentation](https://flask-security-too.readthedocs.io/en/stable/)) @@ -1853,20 +2132,24 @@ The Flask-Security-Too project is provided as open source under the # core.py from datetime import datetime, timedelta +import re +import typing as t import warnings import pkg_resources -~~from flask import _request_ctx_stack, current_app, render_template -from flask_babelex import Domain +~~from flask import _request_ctx_stack, current_app +from flask.json import JSONEncoder from flask_login import AnonymousUserMixin, LoginManager from flask_login import UserMixin as BaseUserMixin from flask_login import current_user from flask_principal import Identity, Principal, RoleNeed, UserNeed, identity_loaded +from flask_wtf import FlaskForm from itsdangerous import URLSafeTimedSerializer from passlib.context import CryptContext from werkzeug.datastructures import ImmutableList from werkzeug.local import LocalProxy +from .babel import FsDomain from .decorators import ( default_reauthn_handler, default_unauthn_handler, @@ -1879,76 +2162,11 @@ from .forms import ( LoginForm, PasswordlessLoginForm, RegisterForm, - ResetPasswordForm, - SendConfirmationForm, - - -## ... source file abbreviated to get to current_app examples ... - - - UnifiedSigninSetupValidateForm, - UnifiedVerifyForm, - us_send_security_token, -) -from .totp import Totp -from .utils import _ -from .utils import config_value as cv -from .utils import ( - FsJsonEncoder, - FsPermNeed, - csrf_cookie_handler, - default_want_json, - default_password_validator, - get_config, - get_identity_attribute, - get_identity_attributes, - get_message, - localize_callback, - set_request_attr, - uia_email_mapper, - url_for_security, - verify_and_update_password, -) -from .views import create_blueprint, default_render_json - -~~_security = LocalProxy(lambda: current_app.extensions["security"]) -_datastore = LocalProxy(lambda: _security.datastore) - -AUTHN_MECHANISMS = ("basic", "session", "token") - - -_default_config = { - "BLUEPRINT_NAME": "security", - "CLI_ROLES_NAME": "roles", - "CLI_USERS_NAME": "users", - "URL_PREFIX": None, - "SUBDOMAIN": None, - "FLASH_MESSAGES": True, - "I18N_DOMAIN": "flask_security", - "I18N_DIRNAME": pkg_resources.resource_filename("flask_security", "translations"), - "PASSWORD_HASH": "bcrypt", - "PASSWORD_SALT": None, - "PASSWORD_SINGLE_HASH": { - "django_argon2", - "django_bcrypt_sha256", - "django_pbkdf2_sha256", - "django_pbkdf2_sha1", - "django_bcrypt", - "django_salted_md5", - "django_salted_sha1", ## ... source file abbreviated to get to current_app examples ... - "SEND_CONFIRMATION_TEMPLATE": "security/send_confirmation.html", - "SEND_LOGIN_TEMPLATE": "security/send_login.html", - "VERIFY_TEMPLATE": "security/verify.html", - "TWO_FACTOR_VERIFY_CODE_TEMPLATE": "security/two_factor_verify_code.html", - "TWO_FACTOR_SETUP_TEMPLATE": "security/two_factor_setup.html", - "CONFIRMABLE": False, - "REGISTERABLE": False, - "RECOVERABLE": False, "TRACKABLE": False, "PASSWORDLESS": False, "CHANGEABLE": False, @@ -1961,6 +2179,14 @@ _default_config = { "TWO_FACTOR_AUTHENTICATOR_VALIDITY": 120, "TWO_FACTOR_MAIL_VALIDITY": 300, "TWO_FACTOR_SMS_VALIDITY": 120, + "TWO_FACTOR_ALWAYS_VALIDATE": True, + "TWO_FACTOR_LOGIN_VALIDITY": "30 days", + "TWO_FACTOR_VALIDITY_SALT": "tf-validity-salt", + "TWO_FACTOR_VALIDITY_COOKIE": { + "httponly": True, + "secure": False, + "samesite": "Strict", + }, "CONFIRM_EMAIL_WITHIN": "5 days", "RESET_PASSWORD_WITHIN": "5 days", "LOGIN_WITHOUT_CONFIRMATION": False, @@ -1996,33 +2222,33 @@ _default_config = { ## ... source file abbreviated to get to current_app examples ... - - for key, value in _default_config.items(): - app.config.setdefault("SECURITY_" + key, value) - - for key, value in _default_messages.items(): - app.config.setdefault("SECURITY_MSG_" + key, value) + for key, value in get_config(app).items(): + setattr(self, key.lower(), value) identity_loaded.connect_via(app)(_on_identity_loaded) - self._state = state = _get_state(app, datastore, **kwargs) - if hasattr(datastore, "user_model") and not hasattr( - datastore.user_model, "fs_uniquifier" + if hasattr(self.datastore, "user_model") and not hasattr( + self.datastore.user_model, "fs_uniquifier" ): # pragma: no cover raise ValueError("User model must contain fs_uniquifier as of 4.0.0") - if register_blueprint: - bp = create_blueprint( - app, state, __name__, json_encoder=kwargs["json_encoder_cls"] - ) - app.register_blueprint(bp) - app.context_processor(_context_processor) + for uia in cv("USER_IDENTITY_ATTRIBUTES", app=app): # pragma: no cover + if not isinstance(uia, dict): + raise ValueError( + "SECURITY_USER_IDENTITY_ATTRIBUTES changed semantics" + " in 4.0 - please see release notes." + ) + if len(list(uia.keys())) != 1: + raise ValueError( + "Each element in SECURITY_USER_IDENTITY_ATTRIBUTES" + " must have one and only one key." + ) @app.before_first_request def _register_i18n(): if "_" not in app.jinja_env.globals: -~~ current_app.jinja_env.globals["_"] = state.i18n_domain.gettext -~~ current_app.jinja_env.globals["_fsdomain"] = state.i18n_domain.gettext +~~ current_app.jinja_env.globals["_"] = self.i18n_domain.gettext +~~ current_app.jinja_env.globals["_fsdomain"] = self.i18n_domain.gettext @app.before_first_request def _csrf_init(): @@ -2052,40 +2278,42 @@ _default_config = { ) csrf_cookie = cv("CSRF_COOKIE") - if csrf_cookie and csrf_cookie["key"] and not csrf: + if csrf_cookie and csrf_cookie.get("key", None): +~~ current_app.config["SECURITY_CSRF_COOKIE_NAME"] = csrf_cookie.pop("key") + if cv("CSRF_COOKIE_NAME") and not csrf: raise ValueError( "CSRF_COOKIE defined however CsrfProtect not part of application" ) if csrf: csrf.exempt("flask_security.views.logout") - if csrf_cookie and csrf_cookie["key"]: + if cv("CSRF_COOKIE_NAME"): ~~ current_app.after_request(csrf_cookie_handler) ~~ current_app.config["WTF_CSRF_HEADERS"].append(cv("CSRF_HEADER")) - state._phone_util = state.phone_util_cls(app) - state._mail_util = state.mail_util_cls(app) + self._phone_util = self.phone_util_cls(app) + self._mail_util = self.mail_util_cls(app) + self._password_util = self.password_util_cls(app) + self._username_util = self.username_util_cls(app) + rvre = cv("REDIRECT_VALIDATE_RE", app=app) + if rvre: + self._redirect_validate_re = re.compile(rvre) - app.extensions["security"] = state + if not hasattr(app, "login_manager") or not self.login_manager: + self.login_manager = _get_login_manager(app, self.anonymous_user) - if hasattr(app, "cli"): - from .cli import users, roles + self.remember_token_serializer = _get_serializer(app, "remember") + self.login_serializer = _get_serializer(app, "login") + self.reset_serializer = _get_serializer(app, "reset") + self.confirm_serializer = _get_serializer(app, "confirm") + self.us_setup_serializer = _get_serializer(app, "us_setup") + self.tf_validity_serializer = _get_serializer(app, "two_factor_validity") + self.principal = _get_principal(app) + self.pwd_context = _get_pwd_context(app) + self.hashing_context = _get_hashing_context(app) + self.i18n_domain = FsDomain(app) - if state.cli_users_name: - app.cli.add_command(users, state.cli_users_name) - if state.cli_roles_name: - app.cli.add_command(roles, state.cli_roles_name) - - for newc, oldc in [ - ("SECURITY_SMS_SERVICE", "SECURITY_TWO_FACTOR_SMS_SERVICE"), - ("SECURITY_SMS_SERVICE_CONFIG", "SECURITY_TWO_FACTOR_SMS_SERVICE_CONFIG"), - ("SECURITY_TOTP_SECRETS", "SECURITY_TWO_FACTOR_SECRET"), - ("SECURITY_TOTP_ISSUER", "SECURITY_TWO_FACTOR_URI_SERVICE_NAME"), - ]: - if not app.config.get(newc, None): - app.config[newc] = app.config.get(oldc, None) - - for uia in cv("USER_IDENTITY_ATTRIBUTES", app=app): # pragma: no cover + if cv("USERNAME_ENABLE", app): ## ... source file continues with no further current_app examples... @@ -2093,7 +2321,7 @@ _default_config = { ``` -## Example 14 from Flask-User +## Example 16 from Flask-User [Flask-User](https://github.com/lingthio/Flask-User) ([PyPI information](https://pypi.org/project/Flask-User/) and @@ -2143,6 +2371,7 @@ class UserManager(UserManager__Settings, UserManager__Utils, UserManager__Views) ## ... source file abbreviated to get to current_app examples ... + @app.before_request def advance_session_timeout(): session.permanent = True # Timeout after app.permanent_session_lifetime period session.modified = True # Advance session timeout each time a user visits a page @@ -2167,7 +2396,7 @@ class UserManager(UserManager__Settings, UserManager__Utils, UserManager__Views) def call_or_get(function_or_property): return function_or_property() if callable(function_or_property) else function_or_property - return dict( +~~ return dict( ~~ user_manager=current_app.user_manager, call_or_get=call_or_get, ) @@ -2200,7 +2429,7 @@ class UserManager(UserManager__Settings, UserManager__Utils, UserManager__Views) ``` -## Example 15 from Flask-VueJs-Template +## Example 17 from Flask-VueJs-Template [Flask-VueJs-Template](https://github.com/gtalarico/flask-vuejs-template) ([demo site](https://flask-vuejs-template.herokuapp.com/)) is a minimal [Flask](/flask.html) boilerplate starter project that @@ -2241,7 +2470,7 @@ def index_client(): ``` -## Example 16 from Flasky +## Example 18 from Flasky [Flasky](https://github.com/miguelgrinberg/flasky) is a wonderful example application by [Miguel Grinberg](https://github.com/miguelgrinberg) that he builds @@ -2296,7 +2525,7 @@ def run_migrations_online(): ``` -## Example 17 from indico +## Example 19 from indico [indico](https://github.com/indico/indico) ([project website](https://getindico.io/), [documentation](https://docs.getindico.io/en/stable/installation/) @@ -2328,10 +2557,12 @@ logging.config.fileConfig(config.config_file_name) ~~target_metadata = current_app.extensions['migrate'].db.metadata -def _include_symbol(tablename, schema): - if schema and schema.startswith('plugin_'): +def _include_object(object_, name, type_, reflected, compare_to): + if type_ != 'table': + return True + if object_.schema and object_.schema.startswith('plugin_'): return False - return tablename != 'alembic_version' and not tablename.startswith('alembic_version_') + return name != 'alembic_version' and not name.startswith('alembic_version_') def _render_item(type_, obj, autogen_context): @@ -2346,18 +2577,16 @@ def _render_item(type_, obj, autogen_context): def run_migrations_offline(): url = config.get_main_option('sqlalchemy.url') context.configure(url=url, target_metadata=target_metadata, include_schemas=True, - version_table_schema='public', include_symbol=_include_symbol, render_item=_render_item, + version_table_schema='public', include_object=_include_object, render_item=_render_item, template_args={'toplevel_code': set()}) - with context.begin_transaction(): - ## ... source file continues with no further current_app examples... ``` -## Example 18 from sandman2 +## Example 20 from sandman2 [sandman2](https://github.com/jeffknupp/sandman2) ([project documentation](https://sandman2.readthedocs.io/en/latest/) and @@ -2482,7 +2711,7 @@ def register_model(cls, admin=None): ``` -## Example 19 from tedivms-flask +## Example 21 from tedivms-flask [tedivm's flask starter app](https://github.com/tedivm/tedivms-flask) is a base of [Flask](/flask.html) code and related projects such as [Celery](/celery.html) which provides a template to start your own diff --git a/content/pages/examples/flask/flask-globals-g.markdown b/content/pages/examples/flask/flask-globals-g.markdown index c14142844..6c47941e6 100644 --- a/content/pages/examples/flask/flask-globals-g.markdown +++ b/content/pages/examples/flask/flask-globals-g.markdown @@ -42,14 +42,13 @@ import datetime import json import logging import re -from typing import Dict, List, Set +from typing import Dict, List, Optional, Set, Tuple ~~from flask import g, session, url_for from flask_babel import lazy_gettext as _ from flask_jwt_extended import current_user as current_user_jwt from flask_jwt_extended import JWTManager from flask_login import current_user, LoginManager -from flask_openid import OpenID from werkzeug.security import check_password_hash, generate_password_hash from .api import SecurityApi @@ -69,6 +68,7 @@ from .views import ( RegisterUserModelView, ResetMyPasswordView, ResetPasswordView, + RoleModelView, ## ... source file abbreviated to get to g examples ... @@ -186,12 +186,6 @@ from .views import ( raise NotImplementedError - def add_permission_view_menu(self, permission_name, view_menu_name): - raise NotImplementedError - - def del_permission_view_menu(self, permission_name, view_menu_name, cascade=True): - raise NotImplementedError - def exist_permission_on_views(self, lst, item): raise NotImplementedError @@ -204,6 +198,12 @@ from .views import ( def del_permission_role(self, role, perm_view): raise NotImplementedError + def export_roles(self, path: Optional[str] = None) -> None: + raise NotImplementedError + + def import_roles(self, path: str) -> None: + raise NotImplementedError + def load_user(self, pk): return self.get_user_by_id(int(pk)) @@ -440,7 +440,7 @@ from werkzeug.urls import url_quote_plus from flask_debugtoolbar.compat import iteritems from flask_debugtoolbar.toolbar import DebugToolbar -from flask_debugtoolbar.utils import decode_text +from flask_debugtoolbar.utils import decode_text, gzip_compress, gzip_decompress try: from importlib.metadata import version @@ -541,11 +541,11 @@ from base64 import b64decode from functools import wraps from hashlib import md5 from random import Random, SystemRandom -~~from flask import request, make_response, session, g +~~from flask import request, make_response, session, g, Response from werkzeug.datastructures import Authorization from werkzeug.security import safe_str_cmp -__version__ = '4.1.1dev' +__version__ = '4.2.1dev' class HTTPAuth(object): @@ -606,9 +606,10 @@ class HTTPAuth(object): return login_required_internal def username(self): - if not request.authorization: + auth = self.get_auth() + if not auth: return "" - return request.authorization.username + return auth.username def current_user(self): if hasattr(g, 'flask_httpauth_user'): @@ -642,7 +643,6 @@ class HTTPBasicAuth(HTTPAuth): ## ... source file abbreviated to get to g examples ... - def decorated(*args, **kwargs): selected_auth = None if 'Authorization' in request.headers: try: @@ -657,8 +657,9 @@ class HTTPBasicAuth(HTTPAuth): break if selected_auth is None: selected_auth = self.main_auth - return selected_auth.login_required(role=role)(f)( - *args, **kwargs) + return selected_auth.login_required(role=role, + optional=optional + )(f)(*args, **kwargs) return decorated if f: @@ -678,7 +679,7 @@ class HTTPBasicAuth(HTTPAuth): ## Example 6 from Flask-WTF [Flask-WTF](https://github.com/lepture/flask-wtf) -([project documentation](https://flask-wtf.readthedocs.io/en/stable/) +([project documentation](https://flask-wtf.readthedocs.io/) and [PyPI page](https://pypi.org/project/Flask-WTF/)) provides a bridge between [Flask](/flask.html) and the the @@ -695,6 +696,7 @@ import hashlib import logging import os import warnings +from urllib.parse import urlparse from functools import wraps ~~from flask import Blueprint, current_app, g, request, session @@ -704,7 +706,7 @@ from werkzeug.security import safe_str_cmp from wtforms import ValidationError from wtforms.csrf.core import CSRF -from ._compat import FlaskWTFDeprecationWarning, string_types, urlparse +from ._compat import FlaskWTFDeprecationWarning __all__ = ('generate_csrf', 'validate_csrf', 'CSRFProtect') logger = logging.getLogger(__name__) @@ -781,7 +783,7 @@ def validate_csrf(data, secret_key=None, time_limit=None, token_key=None): class _FlaskFormCSRF(CSRF): def setup_form(self, form): self.meta = form.meta - return super(_FlaskFormCSRF, self).setup_form(form) + return super().setup_form(form) def generate_csrf_token(self, csrf_token_field): return generate_csrf( @@ -805,7 +807,7 @@ class _FlaskFormCSRF(CSRF): raise -class CSRFProtect(object): +class CSRFProtect: def __init__(self, app=None): self._exempt_views = set() @@ -840,7 +842,7 @@ class CSRFProtect(object): if not request.referrer: self._error_response('The referrer header is missing.') - good_referrer = 'https://{0}/'.format(request.host) + good_referrer = f'https://{request.host}/' if not same_origin(request.referrer, good_referrer): self._error_response('The referrer does not match the host.') @@ -853,7 +855,7 @@ class CSRFProtect(object): self._exempt_blueprints.add(view.name) return view - if isinstance(view, string_types): + if isinstance(view, str): view_location = view else: view_location = '.'.join((view.__module__, view.__name__)) @@ -877,127 +879,7 @@ class CSRFProtect(object): ``` -## Example 7 from Flask-Security-Too -[Flask-Security-Too](https://github.com/Flask-Middleware/flask-security/) -([PyPi page](https://pypi.org/project/Flask-Security-Too/) and -[project documentation](https://flask-security-too.readthedocs.io/en/stable/)) -is a maintained fork of the original -[Flask-Security](https://github.com/mattupstate/flask-security) project that -makes it easier to add common security features to [Flask](/flask.html) -web applications. A few of the critical goals of the Flask-Security-Too -project are ensuring JavaScript client-based single-page applications (SPAs) -can work securely with Flask-based backends and that guidance by the -[OWASP](https://owasp.org/) organization is followed by default. - -The Flask-Security-Too project is provided as open source under the -[MIT license](https://github.com/Flask-Middleware/flask-security/blob/master/LICENSE). - -[**Flask-Security-Too / flask_security / utils.py**](https://github.com/Flask-Middleware/flask-security/blob/master/flask_security/./utils.py) - -```python -# utils.py -import abc -import base64 -import datetime -from functools import partial -import hashlib -import hmac -import time -from typing import Dict, List -import warnings -from datetime import timedelta -from urllib.parse import parse_qsl, parse_qs, urlsplit, urlunsplit, urlencode -import urllib.request -import urllib.error - -~~from flask import _request_ctx_stack, current_app, flash, g, request, session, url_for -from flask.json import JSONEncoder -from flask_login import login_user as _login_user -from flask_login import logout_user as _logout_user -from flask_login import current_user -from flask_login import COOKIE_NAME as REMEMBER_COOKIE_NAME -from flask_principal import AnonymousIdentity, Identity, identity_changed, Need -from flask_wtf import csrf -from wtforms import validators, ValidationError -from itsdangerous import BadSignature, SignatureExpired -from speaklater import is_lazy_string -from werkzeug.local import LocalProxy -from werkzeug.datastructures import MultiDict -from .quart_compat import best -from .signals import user_authenticated - -_security = LocalProxy(lambda: current_app.extensions["security"]) - -_datastore = LocalProxy(lambda: _security.datastore) - -_pwd_context = LocalProxy(lambda: _security.pwd_context) - -_hashing_context = LocalProxy(lambda: _security.hashing_context) - -localize_callback = LocalProxy(lambda: _security.i18n_domain.gettext) - - -## ... source file abbreviated to get to g examples ... - - - user.login_count = user.login_count + 1 if user.login_count else 1 - - _datastore.put(user) - - session["fs_cc"] = "set" # CSRF cookie - session["fs_paa"] = time.time() # Primary authentication at - timestamp - - identity_changed.send( - current_app._get_current_object(), identity=Identity(user.fs_uniquifier) - ) - - user_authenticated.send( - current_app._get_current_object(), user=user, authn_via=authn_via - ) - return True - - -def logout_user(): - - for key in ("identity.name", "identity.auth_type", "fs_paa", "fs_gexp"): - session.pop(key, None) - - csrf_field_name = find_csrf_field_name() - if csrf_field_name: - session.pop(csrf_field_name, None) -~~ g.pop(csrf_field_name, None) - session["fs_cc"] = "clear" - identity_changed.send( - current_app._get_current_object(), identity=AnonymousIdentity() - ) - _logout_user() - - -def check_and_update_authn_fresh(within, grace, method=None): - - if method == "basic": - return True - - if within.total_seconds() < 0: - return True - - if "fs_paa" not in session: - return False - - now = datetime.datetime.utcnow() - new_exp = now + grace - grace_ts = int(new_exp.timestamp()) - - fs_gexp = session.get("fs_gexp", None) - if fs_gexp: - - -## ... source file continues with no further g examples... - -``` - - -## Example 8 from Flask-User +## Example 7 from Flask-User [Flask-User](https://github.com/lingthio/Flask-User) ([PyPI information](https://pypi.org/project/Flask-User/) and @@ -1094,7 +976,7 @@ def allow_unconfirmed_email(view_function): ``` -## Example 9 from indico +## Example 8 from indico [indico](https://github.com/indico/indico) ([project website](https://getindico.io/), [documentation](https://docs.getindico.io/en/stable/installation/) @@ -1108,8 +990,6 @@ The code is open sourced under the ```python # config.py -from __future__ import absolute_import, unicode_literals - import ast import codecs import os @@ -1135,7 +1015,6 @@ DEFAULTS = { 'ATTACHMENT_STORAGE': 'default', 'AUTH_PROVIDERS': {}, 'BASE_URL': None, - 'CACHE_BACKEND': 'files', 'CACHE_DIR': '/opt/indico/cache', 'CATEGORY_CLEANUP': {}, 'CELERY_BROKER': None, @@ -1145,12 +1024,13 @@ DEFAULTS = { 'CUSTOMIZATION_DEBUG': False, 'CUSTOMIZATION_DIR': None, 'CUSTOM_COUNTRIES': {}, + 'CUSTOM_LANGUAGES': {}, ## ... source file abbreviated to get to g examples ... -class IndicoConfig(object): +class IndicoConfig: __slots__ = ('_config', '_exc') @@ -1175,12 +1055,18 @@ class IndicoConfig(object): @property def IMAGES_BASE_URL(self): -~~ return 'static/images' if g.get('static_site') else url_parse('{}/images'.format(self.BASE_URL)).path +~~ return 'static/images' if g.get('static_site') else url_parse(f'{self.BASE_URL}/images').path @property def LATEX_ENABLED(self): return bool(self.XELATEX_PATH) + def validate(self): + from indico.core.auth import login_rate_limiter + login_rate_limiter._get_current_object() # fail in case FAILED_LOGIN_RATE_LIMIT invalid + if self.DEFAULT_TIMEZONE not in pytz.all_timezones_set: + raise ValueError(f'Invalid default timezone: {self.DEFAULT_TIMEZONE}') + def __getattr__(self, name): try: return self.data[name] @@ -1194,8 +1080,6 @@ class IndicoConfig(object): raise AttributeError('cannot change config at runtime') -config = IndicoConfig() - ## ... source file continues with no further g examples... @@ -1203,7 +1087,7 @@ config = IndicoConfig() ``` -## Example 10 from tedivms-flask +## Example 9 from tedivms-flask [tedivm's flask starter app](https://github.com/tedivm/tedivms-flask) is a base of [Flask](/flask.html) code and related projects such as [Celery](/celery.html) which provides a template to start your own diff --git a/content/pages/examples/flask/flask-globals-request.markdown b/content/pages/examples/flask/flask-globals-request.markdown index 97cf9cda4..be097b278 100644 --- a/content/pages/examples/flask/flask-globals-request.markdown +++ b/content/pages/examples/flask/flask-globals-request.markdown @@ -129,11 +129,17 @@ scenarios. CTFd is open sourced under the ```python # test_themes.py -~~from flask import request +import os +import shutil + +import pytest +~~from flask import render_template, render_template_string, request +from jinja2.exceptions import TemplateNotFound from jinja2.sandbox import SecurityError from werkzeug.test import Client -from CTFd.utils import get_config +from CTFd.config import TestingConfig +from CTFd.utils import get_config, set_config from tests.helpers import create_ctfd, destroy_ctfd, gen_user, login_as_user @@ -152,8 +158,6 @@ def test_themes_run_in_sandbox(): def test_themes_cant_access_configpy_attributes(): - app = create_ctfd() - with app.app_context(): ## ... source file abbreviated to get to request examples ... @@ -202,6 +206,28 @@ def test_that_request_path_hijacking_works_properly(): destroy_ctfd(app) +def test_theme_fallback_config(): + + class ThemeFallbackConfig(TestingConfig): + THEME_FALLBACK = False + + app = create_ctfd(config=ThemeFallbackConfig) + try: + os.mkdir(os.path.join(app.root_path, "themes", "foo_fallback")) + except OSError: + pass + + with app.app_context(): + app.config["THEME_FALLBACK"] = False + set_config("ctf_theme", "foo_fallback") + assert app.config["THEME_FALLBACK"] == False + with app.test_client() as client: + try: + r = client.get("/") + except TemplateNotFound: + pass + try: + ## ... source file continues with no further request examples... @@ -346,8 +372,8 @@ from flaskbb.plugins.utils import validate_plugin from flaskbb.user.models import Group, Guest, User from flaskbb.utils.forms import populate_settings_dict, populate_settings_form from flaskbb.utils.helpers import (get_online_users, register_view, - render_template, time_diff, time_utcnow, - FlashAndRedirect) + render_template, redirect_or_next, + time_diff, time_utcnow, FlashAndRedirect) from flaskbb.utils.requirements import (CanBanUser, CanEditUser, IsAdmin, IsAtleastModerator, @@ -447,9 +473,14 @@ class DeleteUser(MethodView): ] def post(self, user_id=None): -~~ if request.is_xhr: -~~ ids = request.get_json()["ids"] - +~~ if request.get_json() is not None: +~~ ids = request.get_json().get("ids") + if not ids: + return jsonify( + message="No ids provided.", + category="error", + status=404 + ) data = [] for user in User.query.filter(User.id.in_(ids)).all(): if current_user.id == user.id: @@ -467,12 +498,7 @@ class DeleteUser(MethodView): ) return jsonify( - message="{} users deleted.".format(len(data)), - category="success", - data=data, - status=200 - ) - + message=f"{len(data)} users deleted.", ## ... source file abbreviated to get to request examples ... @@ -561,8 +587,14 @@ class BanUser(MethodView): ) return redirect(url_for("management.overview")) -~~ if request.is_xhr: -~~ ids = request.get_json()["ids"] +~~ if request.get_json() is not None: +~~ ids = request.get_json().get("ids") + if not ids: + return jsonify( + message="No ids provided.", + category="error", + status=404 + ) data = [] users = User.query.filter(User.id.in_(ids)).all() @@ -573,26 +605,20 @@ class BanUser(MethodView): continue elif user.ban(): - data.append( - { - "id": - user.id, - "type": - "ban", - "reverse": - "unban", - "reverse_name": - _("Unban"), - "reverse_url": - url_for("management.unban_user", user_id=user.id) - } - ) + data.append({ + "id": user.id, + "type": "ban", + "reverse": "unban", + "reverse_name": _("Unban"), + "reverse_url": url_for("management.unban_user", user_id=user.id) + }) + ## ... source file abbreviated to get to request examples ... - return redirect(url_for("management.banned_users")) + return redirect_or_next(url_for("management.banned_users")) class UnbanUser(MethodView): @@ -617,8 +643,14 @@ class UnbanUser(MethodView): ) return redirect(url_for("management.overview")) -~~ if request.is_xhr: -~~ ids = request.get_json()["ids"] +~~ if request.get_json() is not None: +~~ ids = request.get_json().get("ids") + if not ids: + return jsonify( + message="No ids provided.", + category="error", + status=404 + ) data = [] for user in User.query.filter(User.id.in_(ids)).all(): @@ -626,7 +658,7 @@ class UnbanUser(MethodView): data.append( { "id": user.id, - "type": "unban", + "type": "ban", "reverse": "ban", "reverse_name": _("Ban"), "reverse_url": url_for("management.ban_user", @@ -635,11 +667,13 @@ class UnbanUser(MethodView): ) return jsonify( - message="{} users unbanned.".format(len(data)), + message=f"{len(data)} users unbanned.", category="success", - data=data, - status=200 - ) + + +## ... source file abbreviated to get to request examples ... + + user = User.query.filter_by(id=user_id).first_or_404() @@ -648,7 +682,7 @@ class UnbanUser(MethodView): else: flash(_("Could not unban user."), "danger") - return redirect(url_for("management.banned_users")) + return redirect_or_next(url_for("management.users")) class Groups(MethodView): @@ -720,8 +754,15 @@ class DeleteGroup(MethodView): ] def post(self, group_id=None): -~~ if request.is_xhr: -~~ ids = request.get_json()["ids"] +~~ if request.get_json() is not None: +~~ ids = request.get_json().get("ids") + if not ids: + return jsonify( + message="No ids provided.", + category="error", + status=404 + ) + if not (set(ids) & set(["1", "2", "3", "4", "5", "6"])): data = [] for group in Group.query.filter(Group.id.in_(ids)).all(): @@ -739,13 +780,6 @@ class DeleteGroup(MethodView): return jsonify( message="{} groups deleted.".format(len(data)), category="success", - data=data, - status=200 - ) - return jsonify( - message=_("You cannot delete one of the standard groups."), - category="danger", - data=None, ## ... source file abbreviated to get to request examples ... @@ -820,8 +854,14 @@ class MarkReportRead(MethodView): def post(self, report_id=None): -~~ if request.is_xhr: -~~ ids = request.get_json()["ids"] +~~ if request.get_json() is not None: +~~ ids = request.get_json().get("ids") + if not ids: + return jsonify( + message="No ids provided.", + category="error", + status=404 + ) data = [] for report in Report.query.filter(Report.id.in_(ids)).all(): @@ -840,17 +880,12 @@ class MarkReportRead(MethodView): return jsonify( message="{} reports marked as read.".format(len(data)), - category="success", - data=data, - status=200 - ) - - if report_id: ## ... source file abbreviated to get to request examples ... + for report in reports: report.zapped_by = current_user.id report.zapped = time_utcnow() report_list.append(report) @@ -859,7 +894,7 @@ class MarkReportRead(MethodView): db.session.commit() flash(_("All reports were marked as read."), "success") - return redirect(url_for("management.reports")) + return redirect_or_next(url_for("management.reports")) class DeleteReport(MethodView): @@ -875,11 +910,16 @@ class DeleteReport(MethodView): ] def post(self, report_id=None): +~~ if request.get_json() is not None: +~~ ids = request.get_json().get("ids") + if not ids: + return jsonify( + message="No ids provided.", + category="error", + status=404 + ) -~~ if request.is_xhr: -~~ ids = request.get_json()["ids"] data = [] - for report in Report.query.filter(Report.id.in_(ids)).all(): if report.delete(): data.append( @@ -896,12 +936,6 @@ class DeleteReport(MethodView): message="{} reports deleted.".format(len(data)), category="success", data=data, - status=200 - ) - - report = Report.query.filter_by(id=report_id).first_or_404() - report.delete() - flash(_("Report deleted."), "success") ## ... source file continues with no further request examples... @@ -1075,7 +1109,7 @@ from werkzeug.urls import url_quote_plus from flask_debugtoolbar.compat import iteritems from flask_debugtoolbar.toolbar import DebugToolbar -from flask_debugtoolbar.utils import decode_text +from flask_debugtoolbar.utils import decode_text, gzip_compress, gzip_decompress try: from importlib.metadata import version @@ -1187,8 +1221,8 @@ def replace_insensitive(string, target, replacement): response.headers['content-type'].startswith('text/html')): return response - response_html = response.data.decode(response.charset) - + if 'gzip' in response.headers.get('Content-Encoding', ''): + response_html = gzip_decompress(response.data).decode(response.charset) ## ... source file continues with no further request examples... @@ -1312,11 +1346,11 @@ from base64 import b64decode from functools import wraps from hashlib import md5 from random import Random, SystemRandom -~~from flask import request, make_response, session, g +~~from flask import request, make_response, session, g, Response from werkzeug.datastructures import Authorization from werkzeug.security import safe_str_cmp -__version__ = '4.1.1dev' +__version__ = '4.2.1dev' class HTTPAuth(object): @@ -1342,7 +1376,6 @@ class HTTPAuth(object): ## ... source file abbreviated to get to request examples ... - return f def get_user_roles(self, f): self.get_user_roles_callback = f @@ -1352,8 +1385,9 @@ class HTTPAuth(object): @wraps(f) def decorated(*args, **kwargs): res = f(*args, **kwargs) + check_status_code = not isinstance(res, (tuple, Response)) res = make_response(res) - if res.status_code == 200: + if check_status_code and res.status_code == 200: res.status_code = 401 if 'WWW-Authenticate' not in res.headers.keys(): res.headers['WWW-Authenticate'] = self.authenticate_header() @@ -1458,9 +1492,10 @@ class HTTPAuth(object): return login_required_internal def username(self): -~~ if not request.authorization: + auth = self.get_auth() + if not auth: return "" -~~ return request.authorization.username + return auth.username def current_user(self): if hasattr(g, 'flask_httpauth_user'): @@ -1492,9 +1527,14 @@ class HTTPBasicAuth(HTTPAuth): username, password = b64decode(credentials).split(b':', 1) except (ValueError, TypeError): return None + try: + username = username.decode('utf-8') + password = password.decode('utf-8') + except UnicodeDecodeError: + username = None + password = None return Authorization( - scheme, {'username': username.decode('utf-8'), - 'password': password.decode('utf-8')}) + scheme, {'username': username, 'password': password}) def authenticate(self, auth, stored_password): if auth: @@ -1506,11 +1546,6 @@ class HTTPBasicAuth(HTTPAuth): if self.verify_password_callback: return self.verify_password_callback(username, client_password) if not auth: - return - if self.hash_password_callback: - try: - client_password = self.hash_password_callback(client_password) - except TypeError: ## ... source file abbreviated to get to request examples ... @@ -1572,9 +1607,11 @@ class MultiAuth(object): self.main_auth = main_auth self.additional_auth = args - def login_required(self, f=None, role=None): - if f is not None and role is not None: # pragma: no cover - raise ValueError('role is the only supported argument') + def login_required(self, f=None, role=None, optional=None): + if f is not None and \ + (role is not None or optional is not None): # pragma: no cover + raise ValueError( + 'role and optional are the only supported arguments') def login_required_internal(f): @wraps(f) @@ -1593,8 +1630,9 @@ class MultiAuth(object): break if selected_auth is None: selected_auth = self.main_auth - return selected_auth.login_required(role=role)(f)( - *args, **kwargs) + return selected_auth.login_required(role=role, + optional=optional + )(f)(*args, **kwargs) return decorated if f: @@ -1959,7 +1997,7 @@ class marshal_with_field(object): ## Example 12 from Flask-WTF [Flask-WTF](https://github.com/lepture/flask-wtf) -([project documentation](https://flask-wtf.readthedocs.io/en/stable/) +([project documentation](https://flask-wtf.readthedocs.io/) and [PyPI page](https://pypi.org/project/Flask-WTF/)) provides a bridge between [Flask](/flask.html) and the the @@ -1976,6 +2014,7 @@ import hashlib import logging import os import warnings +from urllib.parse import urlparse from functools import wraps ~~from flask import Blueprint, current_app, g, request, session @@ -1985,7 +2024,7 @@ from werkzeug.security import safe_str_cmp from wtforms import ValidationError from wtforms.csrf.core import CSRF -from ._compat import FlaskWTFDeprecationWarning, string_types, urlparse +from ._compat import FlaskWTFDeprecationWarning __all__ = ('generate_csrf', 'validate_csrf', 'CSRFProtect') logger = logging.getLogger(__name__) @@ -2043,7 +2082,7 @@ def generate_csrf(secret_key=None, token_key=None): return view = app.view_functions.get(request.endpoint) - dest = '{0}.{1}'.format(view.__module__, view.__name__) + dest = f'{view.__module__}.{view.__name__}' if dest in self._exempt_views: return @@ -2086,7 +2125,7 @@ def generate_csrf(secret_key=None, token_key=None): ~~ if not request.referrer: self._error_response('The referrer header is missing.') - good_referrer = 'https://{0}/'.format(request.host) + good_referrer = f'https://{request.host}/' if not same_origin(request.referrer, good_referrer): self._error_response('The referrer does not match the host.') @@ -2099,7 +2138,7 @@ def generate_csrf(secret_key=None, token_key=None): self._exempt_blueprints.add(view.name) return view - if isinstance(view, string_types): + if isinstance(view, str): view_location = view else: view_location = '.'.join((view.__module__, view.__name__)) @@ -2120,7 +2159,7 @@ def generate_csrf(secret_key=None, token_key=None): starter project to build a software-as-a-service (SaaS) web application in [Flask](/flask.html), with [Stripe](/stripe.html) for billing. The boilerplate relies on many common Flask extensions such as -[Flask-WTF](https://flask-wtf.readthedocs.io/en/latest/), +[Flask-WTF](https://flask-wtf.readthedocs.io/), [Flask-Login](https://flask-login.readthedocs.io/en/latest/), [Flask-Admin](https://flask-admin.readthedocs.io/en/latest/), and many others. The project is provided as open source under the @@ -2187,11 +2226,11 @@ The Flask-Security-Too project is provided as open source under the # forms.py import inspect +import typing as t ~~from flask import Markup, current_app, request from flask_login import current_user from flask_wtf import FlaskForm as BaseForm -from speaklater import is_lazy_string, make_lazy_string from werkzeug.local import LocalProxy from wtforms import ( BooleanField, @@ -2205,23 +2244,24 @@ from wtforms import ( validators, ) +try: # pragma: no cover + from wtforms.fields import EmailField +except ImportError: + from wtforms.fields.html5 import EmailField +from wtforms.validators import StopValidation + +from .babel import is_lazy_string, make_lazy_string from .confirmable import requires_confirmation from .utils import ( - _, - _datastore, - config_value, - do_flash, - find_user, - get_identity_attribute, ## ... source file abbreviated to get to request examples ... - -class RegisterFormMixin: submit = SubmitField(get_form_field_label("register")) + username: t.ClassVar[Field] + def to_dict(self, only_user): def is_field_and_user_attr(member): @@ -2259,37 +2299,37 @@ class ForgotPasswordForm(Form, UserEmailFormMixin): submit = SubmitField(get_form_field_label("recover_password")) + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self.requires_confirmation = False + def validate(self): if not super().validate(): return False if not self.user.is_active: self.email.errors.append(get_message("DISABLED_ACCOUNT")[0]) return False - if requires_confirmation(self.user): - self.email.errors.append(get_message("CONFIRMATION_REQUIRED")[0]) - return False - return True ## ... source file abbreviated to get to request examples ... - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - - def validate(self): - if not super().validate(): - return False - if not self.user.is_active: self.email.errors.append(get_message("DISABLED_ACCOUNT")[0]) return False return True +login_email_field = EmailField( + get_form_field_label("email"), validators=[email_required] +) + +login_string_field = StringField( + get_form_field_label("email"), validators=[email_required] +) + + class LoginForm(Form, NextFormMixin): - email = StringField(get_form_field_label("email"), validators=[email_required]) password = PasswordField( get_form_field_label("password"), validators=[password_required] ) @@ -2300,7 +2340,7 @@ class LoginForm(Form, NextFormMixin): super().__init__(*args, **kwargs) if not self.next.data: ~~ self.next.data = request.args.get("next", "") - self.remember.default = config_value("DEFAULT_REMEMBER_ME") + self.remember.default = cv("DEFAULT_REMEMBER_ME") if ( current_app.extensions["security"].recoverable and not self.password.description @@ -2312,6 +2352,7 @@ class LoginForm(Form, NextFormMixin): ) ) self.password.description = html + self.requires_confirmation = False def validate(self): if not super().validate(): @@ -2323,7 +2364,6 @@ class LoginForm(Form, NextFormMixin): self.email.errors.append(get_message("USER_DOES_NOT_EXIST")[0]) hash_password(self.password.data) return False - if not self.user.password: ## ... source file abbreviated to get to request examples ... @@ -2343,7 +2383,7 @@ class RegisterForm(ConfirmRegisterForm, NextFormMixin): def validate(self): if not super().validate(): return False - if not config_value("UNIFIED_SIGNIN"): + if not cv("UNIFIED_SIGNIN"): if not self.password_confirm.data or not self.password_confirm.data.strip(): self.password_confirm.errors.append( get_message("PASSWORD_NOT_PROVIDED")[0] @@ -2365,7 +2405,7 @@ class ResetPasswordForm(Form, NewPasswordFormMixin, PasswordConfirmFormMixin): if not super().validate(): return False - pbad = _security._password_validator( + pbad, self.password.data = _security._password_util.validate( self.password.data, False, user=current_user ) if pbad: @@ -2437,8 +2477,8 @@ def on_disconnect(): disconnected = '/' -@socketio.on('connect', namespace='/test') -def on_connect_test(): +@socketio.event(namespace='/test') +def connect(): send('connected-test') send(json.dumps(request.args.to_dict(flat=False))) ~~ send(json.dumps({h: request.headers[h] for h in request.headers.keys() @@ -2451,8 +2491,8 @@ def on_disconnect_test(): disconnected = '/test' -@socketio.on('message') -def on_message(message): +@socketio.event +def message(message): send(message) if message == 'test session': session['a'] = 'b' @@ -2497,6 +2537,7 @@ def get_request_event2(data): ~~ request_event_data = request.event emit('my custom response', data) + socketio.on_event('yet another custom event', get_request_event2) @@ -2508,6 +2549,7 @@ def on_custom_event_test(data): def on_custom_event_test2(data): emit('my custom namespace response', data, namespace='/test') + socketio.on_event('yet another custom namespace event', on_custom_event_test2, namespace='/test') @@ -2517,8 +2559,6 @@ def on_custom_event_broadcast(data): emit('my custom response', data, broadcast=True) -@socketio.on('my custom broadcast namespace event', namespace='/test') -def on_custom_event_broadcast_test(data): ## ... source file abbreviated to get to request examples ... @@ -2536,7 +2576,7 @@ def error_handler_default(value): error_testing_default = True else: raise value - return value + return 'error/default' @socketio.on("error testing", namespace='/unused_namespace') @@ -2723,45 +2763,44 @@ The code is open sourced under the ```python # auth.py -from __future__ import unicode_literals +import functools ~~from flask import current_app, request from flask_multipass import InvalidCredentials, Multipass, NoSuchUser +from werkzeug.local import LocalProxy +from indico.core.config import config +from indico.core.limiter import make_rate_limiter from indico.core.logger import Logger -try: - from flask_multipass.providers.oauth import OAuthInvalidSessionState -except ImportError: - OAuthInvalidSessionState = None - - logger = Logger.get('auth') +login_rate_limiter = LocalProxy(functools.cache(lambda: make_rate_limiter('login', config.FAILED_LOGIN_RATE_LIMIT))) class IndicoMultipass(Multipass): @property def default_local_auth_provider(self): - return next((p for p in self.auth_providers.itervalues() if not p.is_external and p.settings.get('default')), + return next((p for p in self.auth_providers.values() if not p.is_external and p.settings.get('default')), None) @property def sync_provider(self): - return next((p for p in self.identity_providers.itervalues() if p.settings.get('synced_fields')), None) + return next((p for p in self.identity_providers.values() if p.settings.get('synced_fields')), None) + @property + def synced_fields(self): ## ... source file abbreviated to get to request examples ... - self._check_default_provider() def _check_default_provider(self): - sync_providers = [p for p in self.identity_providers.itervalues() if p.settings.get('synced_fields')] + sync_providers = [p for p in self.identity_providers.values() if p.settings.get('synced_fields')] if len(sync_providers) > 1: raise ValueError('There can only be one sync provider.') - auth_providers = self.auth_providers.values() + auth_providers = list(self.auth_providers.values()) external_providers = [p for p in auth_providers if p.is_external] local_providers = [p for p in auth_providers if not p.is_external] if any(p.settings.get('default') for p in external_providers): @@ -2779,14 +2818,17 @@ class IndicoMultipass(Multipass): def handle_auth_error(self, exc, redirect_to_login=False): if isinstance(exc, (NoSuchUser, InvalidCredentials)): + login_rate_limiter.hit() logger.warning('Invalid credentials (ip=%s, provider=%s): %s', ~~ request.remote_addr, exc.provider.name if exc.provider else None, exc) else: + exc_str = str(exc) fn = logger.error - if OAuthInvalidSessionState is not None and isinstance(exc, OAuthInvalidSessionState): + if exc_str.startswith('mismatching_state:'): fn = logger.debug - fn('Authentication via %s failed: %s (%r)', exc.provider.name if exc.provider else None, exc, exc.details) - return super(IndicoMultipass, self).handle_auth_error(exc, redirect_to_login=redirect_to_login) + fn('Authentication via %s failed: %s (%r)', exc.provider.name if exc.provider else None, exc_str, + exc.details) + return super().handle_auth_error(exc, redirect_to_login=redirect_to_login) multipass = IndicoMultipass() @@ -2833,7 +2875,8 @@ from util import base64_to_pil app = Flask(__name__) -from keras.applications.mobilenet_v2 import MobileNetV2 + +from tensorflow.keras.applications.mobilenet_v2 import MobileNetV2 model = MobileNetV2(weights='imagenet') print('Model loaded. Check http://127.0.0.1:5000/') @@ -2907,6 +2950,7 @@ import configparser import json import logging import os +from typing import Union import requests import requests_cache @@ -2932,6 +2976,8 @@ requests_cache.install_cache(cache_name='news_cache', expire_after=300) APP = Flask(__name__) +SESSION = requests.Session() +SESSION.headers.update({'Authorization': API_KEY}) @APP.route('/', methods=['GET', 'POST']) @@ -2956,14 +3002,13 @@ def category(category): country = get_cookie('country') if country is not None: params.update({'country': country}) - response = requests.get(TOP_HEADLINES, - params=params, - headers={'Authorization': API_KEY}) + response = SESSION.get(TOP_HEADLINES, params=params) if response.status_code == 200: pages = count_pages(response.json()) if page > pages: page = pages - return redirect(url_for('category', category=category, page=page)) + return redirect( + url_for('category', category=category, page=page)) articles = parse_articles(response.json()) return render(articles, page, pages, country, category) elif response.status_code == 401: @@ -2972,7 +3017,7 @@ def category(category): @APP.route('/search/', methods=['GET', 'POST']) -def search(query): +def search(query: str): ~~ page = request.args.get('page', default=1, type=int) if page < 1: return redirect(url_for('search', query=query, page=1)) @@ -2984,9 +3029,7 @@ def search(query): } ~~ if request.method == 'POST': return do_post(page, category='search', current_query=query) - response = requests.get(EVERYTHING, - params=params, - headers={'Authorization': API_KEY}) + response = SESSION.get(EVERYTHING, params=params) pages = count_pages(response.json()) if page > pages: page = pages @@ -3020,7 +3063,7 @@ def do_post(page, category='general', current_query=None): return redirect(url_for('category', category=category, page=page)) -def parse_articles(response): +def parse_articles(response: dict) -> list: parsed_articles = [] if response.get('status') == 'ok': for article in response.get('articles'): @@ -3033,16 +3076,16 @@ def parse_articles(response): ## ... source file abbreviated to get to request examples ... + 'source': article['source']['name'] }) return parsed_articles -def count_pages(response): - pages = 0 +def count_pages(response: dict) -> int: if response.get('status') == 'ok': - pages = (-(-response.get('totalResults', 0) // PAGE_SIZE)) - return pages + return (-(-response.get('totalResults', 0) // PAGE_SIZE)) + return 0 def render(articles, page, pages, country, category): @@ -3057,9 +3100,8 @@ def render(articles, page, pages, country, category): pages=pages) -def get_cookie(key): -~~ cookie_value = request.cookies.get(key) - return cookie_value +def get_cookie(key: str) -> Union[str, None]: +~~ return request.cookies.get(key) if __name__ == '__main__': @@ -3460,6 +3502,8 @@ from core.db import Database import os import sys import platform +import urllib +import requests from multiprocessing import Process trape = core.stats.trape @@ -3470,13 +3514,11 @@ db = Database() class victim_server(object): @app.route("/" + trape.victim_path) def homeVictim(): - opener = urllib2.build_opener() - headers = victim_headers(request.user_agent) - opener.addheaders = headers + r = requests.get(trape.url_to_clone, headers=victim_headers2(request.user_agent)) if (trape.type_lure == 'local'): html = assignScripts(victim_inject_code(render_template("/" + trape.url_to_clone), 'payload', '/', trape.gmaps, trape.ipinfo)) else: - html = assignScripts(victim_inject_code(opener.open(trape.url_to_clone).read(), 'payload', trape.url_to_clone, trape.gmaps, trape.ipinfo)) + html = assignScripts(victim_inject_code(r.content, 'payload', trape.url_to_clone, trape.gmaps, trape.ipinfo)) return html @app.route("/register", methods=["POST"]) @@ -3569,7 +3611,7 @@ class victim_server(object): ~~ url = request.args.get('url') if url[0:4] != 'http': url = 'http://' + url - opener = urllib2.build_opener() + opener = urllib.request.build_opener() headers = victim_headers(request.user_agent) opener.addheaders = headers html = assignScripts(victim_inject_code(opener.open(url).read(), 'vscript', url, trape.gmaps, trape.ipinfo)) diff --git a/content/pages/examples/flask/flask-globals-session.markdown b/content/pages/examples/flask/flask-globals-session.markdown index 9f8ee050c..baf253747 100644 --- a/content/pages/examples/flask/flask-globals-session.markdown +++ b/content/pages/examples/flask/flask-globals-session.markdown @@ -41,13 +41,13 @@ scenarios. CTFd is open sourced under the import base64 import requests -from flask import Blueprint +from flask import Blueprint, abort from flask import current_app as app ~~from flask import redirect, render_template, request, session, url_for from itsdangerous.exc import BadSignature, BadTimeSignature, SignatureExpired from CTFd.cache import clear_team_session, clear_user_session -from CTFd.models import Teams, Users, db +from CTFd.models import Teams, UserFieldEntries, UserFields, Users, db from CTFd.utils import config, email, get_app_config, get_config from CTFd.utils import user as current_user from CTFd.utils import validators @@ -73,13 +73,6 @@ auth = Blueprint("auth", __name__) ## ... source file abbreviated to get to session examples ... - log("registrations", "[{date}] {ip} - {name} registered with {email}") - db.session.close() - - if is_teams_mode(): - return redirect(url_for("teams.private")) - - return redirect(url_for("challenges.listing")) else: return render_template("register.html", errors=errors) @@ -97,11 +90,18 @@ def login(): user = Users.query.filter_by(name=name).first() if user: + if user.password is None: + errors.append( + "Your account was registered with a 3rd party authentication provider. " + "Please try logging in with a configured authentication provider." + ) + return render_template("login.html", errors=errors) + if user and verify_password(request.form["password"], user.password): ~~ session.regenerate() login_user(user) - log("logins", "[{date}] {ip} - {name} logged in") + log("logins", "[{date}] {ip} - {name} logged in", name=user.name) db.session.close() if request.args.get("next") and validators.is_safe_url( @@ -111,7 +111,11 @@ def login(): return redirect(url_for("challenges.listing")) else: - log("logins", "[{date}] {ip} - submitted invalid password for {name}") + log( + "logins", + "[{date}] {ip} - submitted invalid password for {name}", + name=user.name, + ) errors.append("Your username or password is incorrect") db.session.close() return render_template("login.html", errors=errors) @@ -119,10 +123,6 @@ def login(): log("logins", "[{date}] {ip} - submitted invalid account information") errors.append("Your username or password is incorrect") db.session.close() - return render_template("login.html", errors=errors) - else: - db.session.close() - return render_template("login.html", errors=errors) ## ... source file continues with no further session examples... @@ -153,8 +153,6 @@ import logging ~~from flask import flash, redirect, request, session, url_for from flask_babel import lazy_gettext -from flask_openid import OpenIDResponse, SessionWrapper -from openid.consumer.consumer import CANCEL, Consumer, SUCCESS from .forms import LoginForm_oid, RegisterUserDBForm, RegisterUserOIDForm from .. import const as c @@ -176,6 +174,8 @@ def get_first_last_name(fullname): class BaseRegisterUser(PublicFormView): route_base = "/register" + email_template = "appbuilder/general/security/register_mail.html" + email_subject = lazy_gettext("Account activation") ## ... source file abbreviated to get to session examples ... @@ -229,8 +229,8 @@ class RegisterUserOIDView(BaseRegisterUser): return redirect(self.get_redirect()) def oid_login_handler(self, f, oid): - if request.args.get("openid_complete") != u"yes": - return f(False) + from flask_openid import OpenIDResponse, SessionWrapper + from openid.consumer.consumer import CANCEL, Consumer, SUCCESS ## ... source file continues with no further session examples... @@ -271,7 +271,8 @@ from babel.core import get_locale_identifier from babel.dates import format_date as babel_format_date from babel.dates import format_datetime as babel_format_datetime from babel.dates import format_timedelta as babel_format_timedelta -~~from flask import flash, g, redirect, request, session, url_for +from babel.dates import format_time as babel_format_time +~~from flask import current_app, flash, g, redirect, request, session, url_for from flask_allows import Permission from flask_babelplus import lazy_gettext as _ from flask_login import current_user @@ -282,32 +283,44 @@ from pytz import UTC from werkzeug.local import LocalProxy from werkzeug.utils import ImportStringError, import_string -from flaskbb._compat import (iteritems, range_method, string_types, text_type, - to_bytes, to_unicode) from flaskbb.extensions import babel, redis_store +from flaskbb.utils.http import is_safe_url from flaskbb.utils.settings import flaskbb_config -try: # compat - FileNotFoundError -except NameError: - FileNotFoundError = IOError logger = logging.getLogger(__name__) _punct_re = re.compile(r'[\t !"#$%&\'()*\-/<=>?@\[\\\]^_`{|},.]+') -def slugify(text, delim=u"-"): - text = unidecode.unidecode(text) - result = [] - for word in _punct_re.split(text.lower()): - if word: +def to_bytes(text, encoding="utf-8"): + if isinstance(text, str): + text = text.encode(encoding) + return text + + +## ... source file abbreviated to get to session examples ... + + result.append(word) - return text_type(delim.join(result)) + return str(delim.join(result)) -def redirect_or_next(endpoint, **kwargs): - return redirect(request.args.get("next") or endpoint, **kwargs) +def redirect_url(endpoint, use_referrer=True): + targets = [endpoint] + allowed_hosts = current_app.config["ALLOWED_HOSTS"] + if use_referrer: + targets.insert(0, request.referrer) + for target in targets: + if target and is_safe_url(target, allowed_hosts): + return target + + +def redirect_or_next(endpoint, use_referrer=True): + return redirect( + redirect_url(request.args.get("next"), use_referrer) + or redirect_url(endpoint, use_referrer) + ) def render_template(template, **context): # pragma: no cover @@ -462,11 +475,11 @@ from base64 import b64decode from functools import wraps from hashlib import md5 from random import Random, SystemRandom -~~from flask import request, make_response, session, g +~~from flask import request, make_response, session, g, Response from werkzeug.datastructures import Authorization from werkzeug.security import safe_str_cmp -__version__ = '4.1.1dev' +__version__ = '4.2.1dev' class HTTPAuth(object): @@ -718,7 +731,7 @@ def login_required(func): ## Example 7 from Flask-WTF [Flask-WTF](https://github.com/lepture/flask-wtf) -([project documentation](https://flask-wtf.readthedocs.io/en/stable/) +([project documentation](https://flask-wtf.readthedocs.io/) and [PyPI page](https://pypi.org/project/Flask-WTF/)) provides a bridge between [Flask](/flask.html) and the the @@ -735,6 +748,7 @@ import hashlib import logging import os import warnings +from urllib.parse import urlparse from functools import wraps ~~from flask import Blueprint, current_app, g, request, session @@ -744,7 +758,7 @@ from werkzeug.security import safe_str_cmp from wtforms import ValidationError from wtforms.csrf.core import CSRF -from ._compat import FlaskWTFDeprecationWarning, string_types, urlparse +from ._compat import FlaskWTFDeprecationWarning __all__ = ('generate_csrf', 'validate_csrf', 'CSRFProtect') logger = logging.getLogger(__name__) @@ -832,7 +846,7 @@ def _get_config( starter project to build a software-as-a-service (SaaS) web application in [Flask](/flask.html), with [Stripe](/stripe.html) for billing. The boilerplate relies on many common Flask extensions such as -[Flask-WTF](https://flask-wtf.readthedocs.io/en/latest/), +[Flask-WTF](https://flask-wtf.readthedocs.io/), [Flask-Login](https://flask-login.readthedocs.io/en/latest/), [Flask-Admin](https://flask-admin.readthedocs.io/en/latest/), and many others. The project is provided as open source under the @@ -919,15 +933,19 @@ The Flask-Security-Too project is provided as open source under the ```python # twofactor.py +import typing as t + ~~from flask import current_app as app, redirect, request, session from werkzeug.datastructures import MultiDict -from werkzeug.local import LocalProxy +from .proxies import _security, _datastore from .utils import ( SmsSenderFactory, base_render_json, + check_and_get_token_status, config_value, do_flash, + get_within_delta, login_user, json_error_response, send_mail, @@ -940,8 +958,8 @@ from .signals import ( tf_profile_changed, ) -_security = LocalProxy(lambda: app.extensions["security"]) -_datastore = LocalProxy(lambda: _security.datastore) +if t.TYPE_CHECKING: # pragma: no cover + from flask import Response def tf_clean_session(): @@ -1198,108 +1216,150 @@ is a [Flask](/flask.html)-based web app for event management. The code is open sourced under the [MIT license](https://github.com/indico/indico/blob/master/LICENSE). -[**indico / indico / core / logger.py**](https://github.com/indico/indico/blob/master/indico/core/logger.py) +[**indico / indico / util / i18n.py**](https://github.com/indico/indico/blob/master/indico/util/i18n.py) ```python -# logger.py +# i18n.py -from __future__ import unicode_literals +import ast +import re +from collections import Counter +from contextlib import contextmanager + +from babel import negotiate_locale +from babel.core import LOCALE_ALIASES, Locale +from babel.messages.pofile import read_po +from babel.support import NullTranslations +~~from flask import current_app, g, has_app_context, has_request_context, request, session +from flask_babel import Babel, Domain, get_domain +from flask_pluginengine import current_plugin +from speaklater import is_lazy_string, make_lazy_string +from werkzeug.utils import cached_property -import logging -import logging.config -import logging.handlers -import os -import smtplib -import warnings -from email.mime.text import MIMEText -from email.utils import formatdate -from pprint import pformat +from indico.core.config import config +from indico.util.caching import memoize_request -import yaml -~~from flask import current_app, has_request_context, request, session -from indico.core.config import config -from indico.util.i18n import set_best_lang -from indico.web.util import get_request_info +LOCALE_ALIASES = dict(LOCALE_ALIASES, en='en_GB') +RE_TR_FUNCTION = re.compile(r'''_\("([^"]*)"\)|_\('([^']*)'\)''', re.DOTALL | re.MULTILINE) +babel = Babel() +_use_context = object() -try: - from raven import setup_logging - from raven.contrib.celery import register_logger_signal, register_signal - from raven.contrib.flask import Sentry - from raven.handlers.logging import SentryHandler -except ImportError: - Sentry = object # so we can subclass - has_sentry = False -else: - has_sentry = True +def get_translation_domain(plugin_name=_use_context): + if plugin_name is None: + return get_domain() + else: + plugin = None + if has_app_context(): + from indico.core.plugins import plugin_engine + plugin = plugin_engine.get_plugin(plugin_name) if plugin_name is not _use_context else current_plugin -class AddRequestIDFilter(object): - def filter(self, record): - record.request_id = request.id if has_request_context() else '0' * 16 - return True +## ... source file abbreviated to get to session examples ... -## ... source file abbreviated to get to session examples ... + def weekday(self, daynum, short=True): + return self.days['format']['abbreviated' if short else 'wide'][daynum] + @cached_property + def time_formats(self): + formats = super().time_formats + for k, v in formats.items(): + v.format = v.format.replace(':%(ss)s', '') + return formats + + +def _remove_locale_script(locale): + parts = locale.split('_') # e.g. `en_GB` or `zh_Hans_CN` + return f'{parts[0]}_{parts[-1]}' + + +@babel.localeselector +def set_best_lang(check_session=True): + from indico.core.config import config + + if not has_request_context(): + return 'en_GB' if current_app.config['TESTING'] else config.DEFAULT_LOCALE + elif 'lang' in g: + return g.lang +~~ elif check_session and session.lang is not None: +~~ return session.lang + + all_locales = {_remove_locale_script(loc).lower(): loc for loc in get_all_locales()} + + preferred = [x.replace('-', '_') for x in request.accept_languages.values()] + resolved_lang = negotiate_locale(preferred, list(all_locales), aliases=LOCALE_ALIASES) + + if not resolved_lang: + if current_app.config['TESTING']: + return 'en_GB' + + resolved_lang = config.DEFAULT_LOCALE - if formatter.pop('append_request_info', False): - assert '()' not in formatter - formatter['()'] = RequestInfoFormatter - if config.DB_LOG: - data['loggers']['indico._db'] = {'level': 'DEBUG', 'propagate': False, 'handlers': ['_db']} - data['handlers']['_db'] = {'class': 'logging.handlers.SocketHandler', 'host': '127.0.0.1', 'port': 9020} - if config.CUSTOMIZATION_DEBUG and config.CUSTOMIZATION_DIR: - data['loggers'].setdefault('indico.customization', {})['level'] = 'DEBUG' - logging.config.dictConfig(data) - if config.SENTRY_DSN: - if not has_sentry: - raise Exception('`raven` must be installed to use sentry logging') - init_sentry(app) - - @classmethod - def get(cls, name=None): - if name is None: - name = 'indico' - elif name != 'indico' and not name.startswith('indico.'): - name = 'indico.' + name - return logging.getLogger(name) - - -class IndicoSentry(Sentry): - def get_user_info(self, request): -~~ if not has_request_context() or not session.user: - return None -~~ return {'id': session.user.id, -~~ 'email': session.user.email, -~~ 'name': session.user.full_name} - - def before_request(self, *args, **kwargs): - super(IndicoSentry, self).before_request() - if not has_request_context(): - return - self.client.extra_context({'Endpoint': str(request.url_rule.endpoint) if request.url_rule else None, - 'Request ID': request.id}) - self.client.tags_context({'locale': set_best_lang()}) - - -def init_sentry(app): - sentry = IndicoSentry(wrap_wsgi=False, register_signal=True, logging=False) - sentry.init_app(app) - handler = SentryHandler(sentry.client, level=getattr(logging, config.SENTRY_LOGGING_LEVEL)) - handler.addFilter(BlacklistFilter({'indico.flask', 'celery.redirected'})) - setup_logging(handler) - register_logger_signal(sentry.client) - register_signal(sentry.client) - - -def sentry_log_exception(): try: - sentry = current_app.extensions['sentry'] + resolved_lang = all_locales[resolved_lang.lower()] except KeyError: + return 'en_GB' + + resolved_lang = re.sub(r'^([a-zA-Z]+)_([a-zA-Z]+)$', + lambda m: f'{m.group(1).lower()}_{m.group(2).upper()}', + resolved_lang) + + g.lang = resolved_lang + return resolved_lang + + +@memoize_request +def get_current_locale(): + return IndicoLocale.parse(set_best_lang()) + + +def get_all_locales(): + if babel.app is None: + return {} + else: + missing = object() + languages = {str(t): config.CUSTOM_LANGUAGES.get(str(t), (t.language_name.title(), t.territory_name)) + for t in babel.list_translations() + if config.CUSTOM_LANGUAGES.get(str(t), missing) is not None} + counts = Counter(x[0] for x in languages.values()) + return {code: (name, territory, counts[name] > 1) for code, (name, territory) in languages.items()} + + +def set_session_lang(lang): +~~ session.lang = lang + + +@contextmanager +def session_language(lang): +~~ old_lang = session.lang + + set_session_lang(lang) + yield + set_session_lang(old_lang) + + +def parse_locale(locale): + return IndicoLocale.parse(locale) + + +def extract_node(node, keywords, commentTags, options, parents=[None]): + if isinstance(node, ast.Str) and isinstance(parents[-1], (ast.Assign, ast.Call)): + matches = RE_TR_FUNCTION.findall(node.s) + for m in matches: + line = m[0] or m[1] + yield (node.lineno, '', line.split('\n'), ['old style recursive strings']) + else: + for cnode in ast.iter_child_nodes(node): + yield from extract_node(cnode, keywords, commentTags, options, parents=(parents + [node])) + + +def po_to_json(po_file, locale=None, domain=None): + with open(po_file, 'rb') as f: + po_data = read_po(f, locale=locale, domain=domain) ## ... source file continues with no further session examples... @@ -1330,7 +1390,8 @@ import os import requests import yaml -~~from flask import Flask, session, render_template +from flask import Flask, render_template +~~from flask import session as current_session from flask_mail import Mail from flask_migrate import Migrate, MigrateCommand from flask.sessions import SessionInterface @@ -1392,7 +1453,7 @@ def get_config(): @user_logged_out.connect_via(app) def clear_session(sender, user, **extra): -~~ session.clear() + current_session.clear() def init_celery_service(app): @@ -1410,13 +1471,6 @@ def init_error_handlers(app): @app.errorhandler(403) def error_forbidden(e): - return show_error(403, 'Forbidden') - - @app.errorhandler(404) - def error_pagenotfound(e): - return show_error(404, 'Page not found.') - - @app.errorhandler(500) ## ... source file continues with no further session examples... @@ -1442,7 +1496,7 @@ from threading import Lock from flask_socketio import SocketIO, emit, join_room, rooms, disconnect import core.stats import core.user -from user_objects import attacks_hook_message +from core.user_objects import attacks_hook_message from core.utils import utils from core.db import Database import sys diff --git a/content/pages/examples/flask/flask-helpers-flash.markdown b/content/pages/examples/flask/flask-helpers-flash.markdown index 0adf66d40..346bb3f6b 100644 --- a/content/pages/examples/flask/flask-helpers-flash.markdown +++ b/content/pages/examples/flask/flask-helpers-flash.markdown @@ -126,8 +126,6 @@ import logging ~~from flask import flash, redirect, request, session, url_for from flask_babel import lazy_gettext -from flask_openid import OpenIDResponse, SessionWrapper -from openid.consumer.consumer import CANCEL, Consumer, SUCCESS from .forms import LoginForm_oid, RegisterUserDBForm, RegisterUserOIDForm from .. import const as c @@ -149,6 +147,8 @@ def get_first_last_name(fullname): class BaseRegisterUser(PublicFormView): route_base = "/register" + email_template = "appbuilder/general/security/register_mail.html" + email_subject = lazy_gettext("Account activation") ## ... source file abbreviated to get to flash examples ... @@ -262,6 +262,9 @@ class RegisterUserDBView(BaseRegisterUser): return redirect(self.get_redirect()) def oid_login_handler(self, f, oid): + from flask_openid import OpenIDResponse, SessionWrapper + from openid.consumer.consumer import CANCEL, Consumer, SUCCESS + if request.args.get("openid_complete") != u"yes": return f(False) consumer = Consumer(SessionWrapper(self), oid.store_factory()) @@ -280,9 +283,6 @@ class RegisterUserDBView(BaseRegisterUser): session["oid_resp"] = resp def form_get(self, form): - self.add_form_unique_validations(form) - - def form_post(self, form): ## ... source file continues with no further flash examples... @@ -918,127 +918,7 @@ class LoginManager(object): ``` -## Example 8 from Flask-Security-Too -[Flask-Security-Too](https://github.com/Flask-Middleware/flask-security/) -([PyPi page](https://pypi.org/project/Flask-Security-Too/) and -[project documentation](https://flask-security-too.readthedocs.io/en/stable/)) -is a maintained fork of the original -[Flask-Security](https://github.com/mattupstate/flask-security) project that -makes it easier to add common security features to [Flask](/flask.html) -web applications. A few of the critical goals of the Flask-Security-Too -project are ensuring JavaScript client-based single-page applications (SPAs) -can work securely with Flask-based backends and that guidance by the -[OWASP](https://owasp.org/) organization is followed by default. - -The Flask-Security-Too project is provided as open source under the -[MIT license](https://github.com/Flask-Middleware/flask-security/blob/master/LICENSE). - -[**Flask-Security-Too / flask_security / utils.py**](https://github.com/Flask-Middleware/flask-security/blob/master/flask_security/./utils.py) - -```python -# utils.py -import abc -import base64 -import datetime -from functools import partial -import hashlib -import hmac -import time -from typing import Dict, List -import warnings -from datetime import timedelta -from urllib.parse import parse_qsl, parse_qs, urlsplit, urlunsplit, urlencode -import urllib.request -import urllib.error - -~~from flask import _request_ctx_stack, current_app, flash, g, request, session, url_for -from flask.json import JSONEncoder -from flask_login import login_user as _login_user -from flask_login import logout_user as _logout_user -from flask_login import current_user -from flask_login import COOKIE_NAME as REMEMBER_COOKIE_NAME -from flask_principal import AnonymousIdentity, Identity, identity_changed, Need -from flask_wtf import csrf -from wtforms import validators, ValidationError -from itsdangerous import BadSignature, SignatureExpired -from speaklater import is_lazy_string -from werkzeug.local import LocalProxy -from werkzeug.datastructures import MultiDict -from .quart_compat import best -from .signals import user_authenticated - -_security = LocalProxy(lambda: current_app.extensions["security"]) - -_datastore = LocalProxy(lambda: _security.datastore) - -_pwd_context = LocalProxy(lambda: _security.pwd_context) - -_hashing_context = LocalProxy(lambda: _security.hashing_context) - -localize_callback = LocalProxy(lambda: _security.i18n_domain.gettext) - - -## ... source file abbreviated to get to flash examples ... - - - string = string.encode("utf-8") - return string - - -def hash_data(data): - return _hashing_context.hash(encode_string(data)) - - -def verify_hash(hashed_data, compare_data): - return _hashing_context.verify(encode_string(compare_data), hashed_data) - - -def suppress_form_csrf(): - if get_request_attr("fs_ignore_csrf"): - return {"csrf": False} - if ( - config_value("CSRF_IGNORE_UNAUTH_ENDPOINTS") - and not current_user.is_authenticated - ): - return {"csrf": False} - return {} - - -def do_flash(message, category=None): - if config_value("FLASH_MESSAGES"): -~~ flash(message, category) - - -def get_url(endpoint_or_url, qparams=None): - try: - return transform_url(url_for(endpoint_or_url), qparams) - except Exception: - if _security.redirect_host: - url = transform_url( - endpoint_or_url, qparams, netloc=_security.redirect_host - ) - else: - url = transform_url(endpoint_or_url, qparams) - - return url - - -def slash_url_suffix(url, suffix): - - return url.endswith("/") and ("%s/" % suffix) or ("/%s" % suffix) - - -def transform_url(url, qparams=None, **kwargs): - if not url: - return url - - -## ... source file continues with no further flash examples... - -``` - - -## Example 9 from Flask-User +## Example 8 from Flask-User [Flask-User](https://github.com/lingthio/Flask-User) ([PyPI information](https://pypi.org/project/Flask-User/) and @@ -1511,7 +1391,7 @@ class UserManager__Views(object): ``` -## Example 10 from indico +## Example 9 from indico [indico](https://github.com/indico/indico) ([project website](https://getindico.io/), [documentation](https://docs.getindico.io/en/stable/installation/) @@ -1525,8 +1405,6 @@ The code is open sourced under the ```python # roles.py -from __future__ import unicode_literals - import csv ~~from flask import flash, session @@ -1535,25 +1413,30 @@ from indico.core.errors import UserValueError from indico.modules.events.roles.forms import ImportMembersCSVForm from indico.modules.users import User from indico.util.i18n import _, ngettext -from indico.util.string import to_unicode, validate_email +from indico.util.spreadsheets import csv_text_io_wrapper +from indico.util.string import validate_email from indico.web.flask.templating import get_template_module from indico.web.util import jsonify_data, jsonify_template -class ImportRoleMembersMixin(object): +class ImportRoleMembersMixin: logger = None def import_members_from_csv(self, f): - reader = csv.reader(f.read().splitlines()) - emails = set() + with csv_text_io_wrapper(f) as ftxt: + reader = csv.reader(ftxt.read().splitlines()) + emails = set() for num_row, row in enumerate(reader, 1): if len(row) != 1: raise UserValueError(_('Row {}: malformed CSV data').format(num_row)) - email = to_unicode(row[0]).strip().lower() + email = row[0].strip().lower() + + +## ... source file abbreviated to get to flash examples ... + - if email and not validate_email(email): raise UserValueError(_('Row {row}: invalid email address: {email}').format(row=num_row, email=email)) if email in emails: raise UserValueError(_('Row {}: email address is not unique').format(num_row)) @@ -1573,18 +1456,18 @@ class ImportRoleMembersMixin(object): if form.remove_existing.data: deleted_members = self.role.members - users for member in deleted_members: - self.logger.info('User {} removed from role {} by {}'.format(member, self.role, session.user)) + self.logger.info(f'User {member} removed from role {self.role} by {session.user}') self.role.members = users else: self.role.members |= users for user in new_members: - self.logger.info('User {} added to role {} by {}'.format(user, self.role, session.user)) -~~ flash(ngettext("{} member has been imported.", - "{} members have been imported.", + self.logger.info(f'User {user} added to role {self.role} by {session.user}') +~~ flash(ngettext('{} member has been imported.', + '{} members have been imported.', len(users)).format(len(users)), 'success') if unknown_emails: -~~ flash(ngettext("There is no user with this email address: {}", - "There are no users with these email addresses: {}", +~~ flash(ngettext('There is no user with this email address: {}', + 'There are no users with these email addresses: {}', len(unknown_emails)).format(', '.join(unknown_emails)), 'warning') tpl = get_template_module('events/roles/_roles.html') return jsonify_data(html=tpl.render_role(self.role, collapsed=False, email_button=False)) @@ -1597,7 +1480,7 @@ class ImportRoleMembersMixin(object): ``` -## Example 11 from tedivms-flask +## Example 10 from tedivms-flask [tedivm's flask starter app](https://github.com/tedivm/tedivms-flask) is a base of [Flask](/flask.html) code and related projects such as [Celery](/celery.html) which provides a template to start your own diff --git a/content/pages/examples/flask/flask-helpers-get-root-path.markdown b/content/pages/examples/flask/flask-helpers-get-root-path.markdown index 0284ba378..3b75f3a3e 100644 --- a/content/pages/examples/flask/flask-helpers-get-root-path.markdown +++ b/content/pages/examples/flask/flask-helpers-get-root-path.markdown @@ -33,25 +33,25 @@ The code is open sourced under the ```python # setup.py -from __future__ import unicode_literals - import os import re import shutil import socket +import subprocess import sys from operator import attrgetter +from pathlib import Path from smtplib import SMTP import click from click import wrap_text ~~from flask.helpers import get_root_path +from packaging.specifiers import SpecifierSet +from packaging.version import Version from pkg_resources import iter_entry_points from prompt_toolkit import prompt -from prompt_toolkit.contrib.completers import PathCompleter, WordCompleter -from prompt_toolkit.layout.lexers import SimpleLexer -from prompt_toolkit.styles import style_from_dict -from prompt_toolkit.token import Token +from prompt_toolkit.completion import PathCompleter, WordCompleter +from prompt_toolkit.styles import Style from pytz import all_timezones, common_timezones from redis import RedisError, StrictRedis from sqlalchemy import create_engine @@ -60,18 +60,18 @@ from sqlalchemy.pool import NullPool from terminaltables import AsciiTable from werkzeug.urls import url_parse +import indico from indico.core.db.sqlalchemy.util.models import import_all_models from indico.util.console import cformat from indico.util.string import validate_email -click.disable_unicode_literals_warning = True - - def _echo(msg=''): click.echo(msg, err=True) + + ## ... source file abbreviated to get to get_root_path examples ... @@ -103,25 +103,25 @@ def _get_dirs(target_dir): ~~ return get_root_path('indico'), os.path.abspath(target_dir) -PROMPT_TOOLKIT_STYLE = style_from_dict({ - Token.HELP: '#aaaaaa', - Token.PROMPT: '#5f87ff', - Token.DEFAULT: '#dfafff', - Token.BRACKET: '#ffffff', - Token.COLON: '#ffffff', - Token.INPUT: '#aaffaa', +PROMPT_TOOLKIT_STYLE = Style.from_dict({ + 'help': '#aaaaaa', + 'prompt': '#5f87ff', + 'default': '#dfafff', + 'bracket': '#ffffff', + 'colon': '#ffffff', + '': '#aaffaa', # user input }) def _prompt(message, default='', path=False, list_=None, required=True, validate=None, allow_invalid=False, password=False, help=None): - def _get_prompt_tokens(cli): + def _get_prompt_tokens(): rv = [ - (Token.PROMPT, message), - (Token.COLON, ': '), + ('class:prompt', message), + ('class:colon', ': '), ] if first and help: - rv.insert(0, (Token.HELP, wrap_text(help) + '\n')) + rv.insert(0, ('class:help', wrap_text(help) + '\n')) return rv completer = None @@ -130,17 +130,17 @@ def _prompt(message, default='', path=False, list_=None, required=True, validate ## ... source file abbreviated to get to get_root_path examples ... + 'SMTP_USE_CELERY = False' + ] - if dev: + if not self.system_notices: config_data += [ - b'', - b'# Development options', - b'DB_LOG = True', - b'DEBUG = True', - b'SMTP_USE_CELERY = False' + '', + '# Disable system notices', + 'SYSTEM_NOTICES_URL = None' ] - config = b'\n'.join(x for x in config_data if x is not None) + config = '\n'.join(x for x in config_data if x is not None) if dev: if not os.path.exists(self.data_root_path): @@ -152,8 +152,8 @@ def _prompt(message, default='', path=False, list_=None, required=True, validate os.mkdir(path) _echo(cformat('%{magenta}Creating %{magenta!}{}%{reset}%{magenta}').format(self.config_path)) - with open(self.config_path, 'wb') as f: - f.write(config + b'\n') + with open(self.config_path, 'w') as f: + f.write(config + '\n') ~~ package_root = get_root_path('indico') _copy(os.path.normpath(os.path.join(package_root, 'logging.yaml.sample')), @@ -174,11 +174,7 @@ def _prompt(message, default='', path=False, list_=None, required=True, validate _echo(cformat('Run %{green!}export INDICO_CONFIG={}%{reset} to use your config file') .format(self.config_path)) - if self.old_archive_dir: - _echo(cformat('Check %{green!}https://git.io/vHP6o%{reset} for a guide on how to ' - 'import data from Indico v1.2')) - else: - _echo(cformat('You can now run %{green!}indico db prepare%{reset} to initialize your Indico database')) + _echo(cformat('You can now run %{green!}indico db prepare%{reset} to initialize your Indico database')) diff --git a/content/pages/examples/flask/flask-helpers-make-response.markdown b/content/pages/examples/flask/flask-helpers-make-response.markdown index 2dc887a82..877d38532 100644 --- a/content/pages/examples/flask/flask-helpers-make-response.markdown +++ b/content/pages/examples/flask/flask-helpers-make-response.markdown @@ -150,11 +150,11 @@ from base64 import b64decode from functools import wraps from hashlib import md5 from random import Random, SystemRandom -~~from flask import request, make_response, session, g +~~from flask import request, make_response, session, g, Response from werkzeug.datastructures import Authorization from werkzeug.security import safe_str_cmp -__version__ = '4.1.1dev' +__version__ = '4.2.1dev' class HTTPAuth(object): @@ -187,8 +187,9 @@ class HTTPAuth(object): @wraps(f) def decorated(*args, **kwargs): res = f(*args, **kwargs) + check_status_code = not isinstance(res, (tuple, Response)) ~~ res = make_response(res) - if res.status_code == 200: + if check_status_code and res.status_code == 200: res.status_code = 401 if 'WWW-Authenticate' not in res.headers.keys(): res.headers['WWW-Authenticate'] = self.authenticate_header() @@ -323,20 +324,19 @@ The Flask-Security-Too project is provided as open source under the ```python # views.py +from functools import partial import time +import typing as t from flask import ( Blueprint, - abort, after_this_request, - current_app, jsonify, request, session, ) from flask_login import current_user -from werkzeug.datastructures import MultiDict -from werkzeug.local import LocalProxy +from werkzeug.datastructures import ImmutableMultiDict, MultiDict from .changeable import change_user_password from .confirmable import ( @@ -346,22 +346,28 @@ from .confirmable import ( ) from .decorators import anonymous_user_required, auth_required, unauth_csrf from .passwordless import login_token_status, send_login_instructions +from .proxies import _security, _datastore from .quart_compat import get_quart_status from .unified_signin import ( us_signin, us_signin_send_code, - us_qrcode, us_setup, us_setup_validate, us_verify, us_verify_link, us_verify_send_code, ) +from .recoverable import ( ## ... source file abbreviated to get to make_response examples ... +from .utils import ( + base_render_json, + check_and_update_authn_fresh, + config_value as cv, + do_flash, get_message, get_post_login_redirect, get_post_logout_redirect, @@ -376,26 +382,16 @@ from .unified_signin import ( slash_url_suffix, suppress_form_csrf, url_for_security, + view_commit, ) if get_quart_status(): # pragma: no cover from quart import make_response, redirect - - async def _commit(response=None): - _datastore.commit() - return response - - else: ~~ from flask import make_response, redirect - def _commit(response=None): - _datastore.commit() - return response - - -_security = LocalProxy(lambda: current_app.extensions["security"]) -_datastore = LocalProxy(lambda: _security.datastore) +if t.TYPE_CHECKING: # pragma: no cover + from flask.typing import ResponseValue def default_render_json(payload, code, headers, user): @@ -411,7 +407,7 @@ def _ctx(endpoint): @unauth_csrf(fall_through=True) -def login(): +def login() -> "ResponseValue": if current_user.is_authenticated and request.method == "POST": @@ -421,7 +417,7 @@ def login(): ) return _security._render_json(payload, 400, None, None) else: - return redirect(get_post_login_redirect()) + return redirect(get_url(cv("POST_LOGIN_VIEW"))) form_class = _security.login_form @@ -451,6 +447,7 @@ import configparser import json import logging import os +from typing import Union import requests import requests_cache @@ -476,20 +473,20 @@ requests_cache.install_cache(cache_name='news_cache', expire_after=300) APP = Flask(__name__) +SESSION = requests.Session() +SESSION.headers.update({'Authorization': API_KEY}) -@APP.route('/', methods=['GET', 'POST']) -def root(): ## ... source file abbreviated to get to make_response examples ... + 'pageSize': PAGE_SIZE + } if request.method == 'POST': return do_post(page, category='search', current_query=query) - response = requests.get(EVERYTHING, - params=params, - headers={'Authorization': API_KEY}) + response = SESSION.get(EVERYTHING, params=params) pages = count_pages(response.json()) if page > pages: page = pages @@ -523,7 +520,7 @@ def do_post(page, category='general', current_query=None): return redirect(url_for('category', category=category, page=page)) -def parse_articles(response): +def parse_articles(response: dict) -> list: parsed_articles = [] if response.get('status') == 'ok': for article in response.get('articles'): diff --git a/content/pages/examples/flask/flask-helpers-safe-join.markdown b/content/pages/examples/flask/flask-helpers-safe-join.markdown index 7959e05d2..1382094d8 100644 --- a/content/pages/examples/flask/flask-helpers-safe-join.markdown +++ b/content/pages/examples/flask/flask-helpers-safe-join.markdown @@ -25,76 +25,98 @@ as-is to run CTF events, or modified for custom rules for related scenarios. CTFd is open sourced under the [Apache License 2.0](https://github.com/CTFd/CTFd/blob/master/LICENSE). -[**CTFd / CTFd / views.py**](https://github.com/CTFd/CTFd/blob/master/./CTFd/views.py) +[**CTFd / CTFd / __init__.py**](https://github.com/CTFd/CTFd/blob/master/./CTFd/__init__.py) ```python -# views.py +# __init__.py +import datetime import os +import sys +import weakref +from distutils.version import StrictVersion -from flask import Blueprint, abort -from flask import current_app as app -from flask import redirect, render_template, request, send_file, session, url_for +import jinja2 +from flask import Flask, Request ~~from flask.helpers import safe_join -from sqlalchemy.exc import IntegrityError - -from CTFd.cache import cache -from CTFd.constants.config import ( - AccountVisibilityTypes, - ChallengeVisibilityTypes, - ConfigTypes, - RegistrationVisibilityTypes, - ScoreVisibilityTypes, +from flask_migrate import upgrade +from jinja2 import FileSystemLoader +from jinja2.sandbox import SandboxedEnvironment +from werkzeug.middleware.proxy_fix import ProxyFix +from werkzeug.utils import cached_property + +import CTFd.utils.config +from CTFd import utils +from CTFd.constants.themes import ADMIN_THEME, DEFAULT_THEME +from CTFd.plugins import init_plugins +from CTFd.utils.crypto import sha256 +from CTFd.utils.initialization import ( + init_events, + init_logs, + init_request_processors, + init_template_filters, + init_template_globals, ) -from CTFd.models import ( - Admins, - Files, - Notifications, - Pages, - Teams, - Users, - UserTokens, - db, -) -from CTFd.utils import config, get_config, set_config -from CTFd.utils import user as current_user -from CTFd.utils import validators -from CTFd.utils.config import is_setup - - -## ... source file abbreviated to get to safe_join examples ... +from CTFd.utils.migrations import create_database, migrations, stamp_latest_revision +from CTFd.utils.sessions import CachingSessionInterface +from CTFd.utils.updates import update_check +__version__ = "3.4.0" +__channel__ = "oss" - abort(403) - else: - abort(403) - if team: - if team.banned: - abort(403) - else: - pass +## ... source file abbreviated to get to safe_join examples ... - if file_id != f.id: - abort(403) - except (BadTimeSignature, SignatureExpired, BadSignature): - abort(403) + self.cache[cache_key] = template + return template + + +class ThemeLoader(FileSystemLoader): + + DEFAULT_THEMES_PATH = os.path.join(os.path.dirname(__file__), "themes") + _ADMIN_THEME_PREFIX = ADMIN_THEME + "/" + + def __init__( + self, + searchpath=DEFAULT_THEMES_PATH, + theme_name=None, + encoding="utf-8", + followlinks=False, + ): + super(ThemeLoader, self).__init__(searchpath, encoding, followlinks) + self.theme_name = theme_name + + def get_source(self, environment, template): + if template.startswith(self._ADMIN_THEME_PREFIX): + if self.theme_name != ADMIN_THEME: + raise jinja2.TemplateNotFound(template) + template = template[len(self._ADMIN_THEME_PREFIX) :] + theme_name = self.theme_name or str(utils.get_config("ctf_theme")) +~~ template = safe_join(theme_name, "templates", template) + return super(ThemeLoader, self).get_source(environment, template) + + +def confirm_upgrade(): + if sys.stdin.isatty(): + print("/*\\ CTFd has updated and must update the database! /*\\") + print("/*\\ Please backup your database before proceeding! /*\\") + print("/*\\ CTFd maintainers are not responsible for any data loss! /*\\") + if input("Run database migrations (Y/N)").lower().strip() == "y": # nosec B322 + return True + else: + print("/*\\ Ignored database migrations... /*\\") + return False + else: + return True - uploader = get_uploader() - try: - return uploader.download(f.location) - except IOError: - abort(404) +def run_upgrade(): + upgrade() + utils.set_config("ctf_version", __version__) -@views.route("/themes//static/") -def themes(theme, path): -~~ filename = safe_join(app.root_path, "themes", theme, "static", path) - if os.path.isfile(filename): - return send_file(filename) - else: - abort(404) +def create_app(config="CTFd.config.Config"): + app = CTFdFlask(__name__) ## ... source file continues with no further safe_join examples... diff --git a/content/pages/examples/flask/flask-helpers-send-file.markdown b/content/pages/examples/flask/flask-helpers-send-file.markdown index 3ae76de0e..726a0993c 100644 --- a/content/pages/examples/flask/flask-helpers-send-file.markdown +++ b/content/pages/examples/flask/flask-helpers-send-file.markdown @@ -45,6 +45,7 @@ from flask import Blueprint, abort from flask import current_app as app ~~from flask import redirect, render_template, request, send_file, session, url_for from flask.helpers import safe_join +from jinja2.exceptions import TemplateNotFound from sqlalchemy.exc import IntegrityError from CTFd.cache import cache @@ -55,6 +56,7 @@ from CTFd.constants.config import ( RegistrationVisibilityTypes, ScoreVisibilityTypes, ) +from CTFd.constants.themes import DEFAULT_THEME from CTFd.models import ( Admins, Files, @@ -66,16 +68,11 @@ from CTFd.models import ( db, ) from CTFd.utils import config, get_config, set_config -from CTFd.utils import user as current_user -from CTFd.utils import validators ## ... source file abbreviated to get to send_file examples ... - abort(403) - - if team: if team.banned: abort(403) else: @@ -96,11 +93,13 @@ from CTFd.utils import validators @views.route("/themes//static/") def themes(theme, path): - filename = safe_join(app.root_path, "themes", theme, "static", path) - if os.path.isfile(filename): -~~ return send_file(filename) - else: - abort(404) + for cand_path in ( + safe_join(app.root_path, "themes", cand_theme, "static", path) + for cand_theme in (theme, *config.ctf_theme_candidates()) + ): + if os.path.isfile(cand_path): +~~ return send_file(cand_path) + abort(404) diff --git a/content/pages/examples/flask/flask-helpers-url-for.markdown b/content/pages/examples/flask/flask-helpers-url-for.markdown index 92e4fa985..9dab48b41 100644 --- a/content/pages/examples/flask/flask-helpers-url-for.markdown +++ b/content/pages/examples/flask/flask-helpers-url-for.markdown @@ -224,13 +224,19 @@ from .security.decorators import permission_name, protect class MenuItem(object): - def __init__(self, name, href="", icon="", label="", childs=None, baseview=None): + def __init__( + self, name, href="", icon="", label="", childs=None, baseview=None, cond=None + ): self.name = name self.href = href self.icon = icon self.label = label self.childs = childs or [] self.baseview = baseview + self.cond = cond + + def should_render(self) -> bool: + return bool(self.cond()) if self.cond is not None else True def get_url(self): if not self.href: @@ -291,7 +297,7 @@ import re import mistune ~~from flask import url_for -from jinja2 import Markup +from markupsafe import Markup from pluggy import HookimplMarker from pygments import highlight from pygments.formatters import HtmlFormatter @@ -350,7 +356,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) backend, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling and many others. flask-base is provided as open source under the @@ -759,7 +765,7 @@ def ui_for(api): starter project to build a software-as-a-service (SaaS) web application in [Flask](/flask.html), with [Stripe](/stripe.html) for billing. The boilerplate relies on many common Flask extensions such as -[Flask-WTF](https://flask-wtf.readthedocs.io/en/latest/), +[Flask-WTF](https://flask-wtf.readthedocs.io/), [Flask-Login](https://flask-login.readthedocs.io/en/latest/), [Flask-Admin](https://flask-admin.readthedocs.io/en/latest/), and many others. The project is provided as open source under the @@ -895,127 +901,7 @@ def reset(token): ``` -## Example 11 from Flask-Security-Too -[Flask-Security-Too](https://github.com/Flask-Middleware/flask-security/) -([PyPi page](https://pypi.org/project/Flask-Security-Too/) and -[project documentation](https://flask-security-too.readthedocs.io/en/stable/)) -is a maintained fork of the original -[Flask-Security](https://github.com/mattupstate/flask-security) project that -makes it easier to add common security features to [Flask](/flask.html) -web applications. A few of the critical goals of the Flask-Security-Too -project are ensuring JavaScript client-based single-page applications (SPAs) -can work securely with Flask-based backends and that guidance by the -[OWASP](https://owasp.org/) organization is followed by default. - -The Flask-Security-Too project is provided as open source under the -[MIT license](https://github.com/Flask-Middleware/flask-security/blob/master/LICENSE). - -[**Flask-Security-Too / flask_security / utils.py**](https://github.com/Flask-Middleware/flask-security/blob/master/flask_security/./utils.py) - -```python -# utils.py -import abc -import base64 -import datetime -from functools import partial -import hashlib -import hmac -import time -from typing import Dict, List -import warnings -from datetime import timedelta -from urllib.parse import parse_qsl, parse_qs, urlsplit, urlunsplit, urlencode -import urllib.request -import urllib.error - -~~from flask import _request_ctx_stack, current_app, flash, g, request, session, url_for -from flask.json import JSONEncoder -from flask_login import login_user as _login_user -from flask_login import logout_user as _logout_user -from flask_login import current_user -from flask_login import COOKIE_NAME as REMEMBER_COOKIE_NAME -from flask_principal import AnonymousIdentity, Identity, identity_changed, Need -from flask_wtf import csrf -from wtforms import validators, ValidationError -from itsdangerous import BadSignature, SignatureExpired -from speaklater import is_lazy_string -from werkzeug.local import LocalProxy -from werkzeug.datastructures import MultiDict -from .quart_compat import best -from .signals import user_authenticated - -_security = LocalProxy(lambda: current_app.extensions["security"]) - -_datastore = LocalProxy(lambda: _security.datastore) - -_pwd_context = LocalProxy(lambda: _security.pwd_context) - -_hashing_context = LocalProxy(lambda: _security.hashing_context) - -localize_callback = LocalProxy(lambda: _security.i18n_domain.gettext) - - -## ... source file abbreviated to get to url_for examples ... - - - return url - - -def slash_url_suffix(url, suffix): - - return url.endswith("/") and ("%s/" % suffix) or ("/%s" % suffix) - - -def transform_url(url, qparams=None, **kwargs): - if not url: - return url - link_parse = urlsplit(url) - if qparams: - current_query = dict(parse_qsl(link_parse.query)) - current_query.update(qparams) - link_parse = link_parse._replace(query=urlencode(current_query)) - return urlunsplit(link_parse._replace(**kwargs)) - - -def get_security_endpoint_name(endpoint): - return f"{_security.blueprint_name}.{endpoint}" - - -def url_for_security(endpoint, **values): - endpoint = get_security_endpoint_name(endpoint) -~~ return url_for(endpoint, **values) - - -def validate_redirect_url(url): - if url is None or url.strip() == "": - return False - url_next = urlsplit(url) - url_base = urlsplit(request.host_url) - if (url_next.netloc or url_next.scheme) and url_next.netloc != url_base.netloc: - return False - return True - - -def get_post_action_redirect(config_key, declared=None): - urls = [ - get_url(request.args.get("next", None)), - get_url(request.form.get("next", None)), - find_redirect(config_key), - ] - if declared: - urls.insert(0, declared) - for url in urls: - if validate_redirect_url(url): - return url - - - -## ... source file continues with no further url_for examples... - -``` - - -## Example 12 from Flask-User +## Example 11 from Flask-User [Flask-User](https://github.com/lingthio/Flask-User) ([PyPI information](https://pypi.org/project/Flask-User/) and @@ -1164,7 +1050,7 @@ class EmailManager(object): ``` -## Example 13 from Flasky +## Example 12 from Flasky [Flasky](https://github.com/miguelgrinberg/flasky) is a wonderful example application by [Miguel Grinberg](https://github.com/miguelgrinberg) that he builds @@ -1238,9 +1124,9 @@ class Role(db.Model): def to_json(self): json_user = { ~~ 'url': url_for('api.get_user', id=self.id), - 'username': self.username, - 'member_since': self.member_since, - 'last_seen': self.last_seen, +~~ 'username': self.username, +~~ 'member_since': self.member_since, +~~ 'last_seen': self.last_seen, ~~ 'posts_url': url_for('api.get_user_posts', id=self.id), ~~ 'followed_posts_url': url_for('api.get_user_followed_posts', id=self.id), @@ -1298,9 +1184,9 @@ class Post(db.Model): def to_json(self): json_post = { ~~ 'url': url_for('api.get_post', id=self.id), - 'body': self.body, - 'body_html': self.body_html, - 'timestamp': self.timestamp, +~~ 'body': self.body, +~~ 'body_html': self.body_html, +~~ 'timestamp': self.timestamp, ~~ 'author_url': url_for('api.get_user', id=self.author_id), ~~ 'comments_url': url_for('api.get_post_comments', id=self.id), 'comment_count': self.comments.count() @@ -1340,9 +1226,9 @@ class Comment(db.Model): json_comment = { ~~ 'url': url_for('api.get_comment', id=self.id), ~~ 'post_url': url_for('api.get_post', id=self.post_id), - 'body': self.body, - 'body_html': self.body_html, - 'timestamp': self.timestamp, +~~ 'body': self.body, +~~ 'body_html': self.body_html, +~~ 'timestamp': self.timestamp, ~~ 'author_url': url_for('api.get_user', id=self.author_id), } return json_comment @@ -1364,7 +1250,7 @@ db.event.listen(Comment.body, 'set', Comment.on_changed_body) ``` -## Example 14 from Datadog Flask Example App +## Example 13 from Datadog Flask Example App The [Datadog Flask example app](https://github.com/DataDog/trace-examples/tree/master/python/flask) contains many examples of the [Flask](/flask.html) core functions available to a developer using the [web framework](/web-frameworks.html). diff --git a/content/pages/examples/flask/flask-json-jsonencoder.markdown b/content/pages/examples/flask/flask-json-jsonencoder.markdown index fd4d8ce5d..7c217d86c 100644 --- a/content/pages/examples/flask/flask-json-jsonencoder.markdown +++ b/content/pages/examples/flask/flask-json-jsonencoder.markdown @@ -32,111 +32,43 @@ can work securely with Flask-based backends and that guidance by the The Flask-Security-Too project is provided as open source under the [MIT license](https://github.com/Flask-Middleware/flask-security/blob/master/LICENSE). -[**Flask-Security-Too / flask_security / utils.py**](https://github.com/Flask-Middleware/flask-security/blob/master/flask_security/./utils.py) +[**Flask-Security-Too / flask_security / core.py**](https://github.com/Flask-Middleware/flask-security/blob/master/flask_security/./core.py) ```python -# utils.py -import abc -import base64 -import datetime -from functools import partial -import hashlib -import hmac -import time -from typing import Dict, List +# core.py + +from datetime import datetime, timedelta +import re +import typing as t import warnings -from datetime import timedelta -from urllib.parse import parse_qsl, parse_qs, urlsplit, urlunsplit, urlencode -import urllib.request -import urllib.error -from flask import _request_ctx_stack, current_app, flash, g, request, session, url_for +import pkg_resources +from flask import _request_ctx_stack, current_app ~~from flask.json import JSONEncoder -from flask_login import login_user as _login_user -from flask_login import logout_user as _logout_user +from flask_login import AnonymousUserMixin, LoginManager +from flask_login import UserMixin as BaseUserMixin from flask_login import current_user -from flask_login import COOKIE_NAME as REMEMBER_COOKIE_NAME -from flask_principal import AnonymousIdentity, Identity, identity_changed, Need -from flask_wtf import csrf -from wtforms import validators, ValidationError -from itsdangerous import BadSignature, SignatureExpired -from speaklater import is_lazy_string +from flask_principal import Identity, Principal, RoleNeed, UserNeed, identity_loaded +from flask_wtf import FlaskForm +from itsdangerous import URLSafeTimedSerializer +from passlib.context import CryptContext +from werkzeug.datastructures import ImmutableList from werkzeug.local import LocalProxy -from werkzeug.datastructures import MultiDict -from .quart_compat import best -from .signals import user_authenticated - -_security = LocalProxy(lambda: current_app.extensions["security"]) - -_datastore = LocalProxy(lambda: _security.datastore) - -_pwd_context = LocalProxy(lambda: _security.pwd_context) - -_hashing_context = LocalProxy(lambda: _security.hashing_context) - -localize_callback = LocalProxy(lambda: _security.i18n_domain.gettext) - - - -## ... source file abbreviated to get to JSONEncoder examples ... - - - accept_mimetypes = req.accept_mimetypes - if not hasattr(req.accept_mimetypes, "best"): # pragma: no cover - accept_mimetypes.best = best - if accept_mimetypes.best == "application/json": - return True - return False - - -def json_error_response(errors): - if isinstance(errors, str): - response_json = dict(error=errors) - elif isinstance(errors, dict): - response_json = dict(errors=errors) - else: - raise TypeError("The errors argument should be either a str or dict.") - - return response_json - - -~~class FsJsonEncoder(JSONEncoder): - - def default(self, obj): - if is_lazy_string(obj): - return str(obj) - else: -~~ return JSONEncoder.default(self, obj) - - -class SmsSenderBaseClass(metaclass=abc.ABCMeta): - def __init__(self): - pass - - @abc.abstractmethod - def send_sms(self, from_number, to_number, msg): # pragma: no cover - return - - -class DummySmsSender(SmsSenderBaseClass): - def send_sms(self, from_number, to_number, msg): # pragma: no cover - return - - -class SmsSenderFactory: - senders = {"Dummy": DummySmsSender} - - @classmethod - def createSender(cls, name, *args, **kwargs): - return cls.senders[name](*args, **kwargs) - - - return _security._render_json(payload, code, headers=None, user=user) - -def default_want_json(req): - if req.is_json: - return True +from .babel import FsDomain +from .decorators import ( + default_reauthn_handler, + default_unauthn_handler, + default_unauthz_handler, +) +from .forms import ( + ChangePasswordForm, + ConfirmRegisterForm, + ForgotPasswordForm, + LoginForm, + PasswordlessLoginForm, + RegisterForm, + RegisterFormMixin, ## ... source file continues with no further JSONEncoder examples... diff --git a/content/pages/examples/flask/flask-json-jsonify.markdown b/content/pages/examples/flask/flask-json-jsonify.markdown index 53923a951..035721414 100644 --- a/content/pages/examples/flask/flask-json-jsonify.markdown +++ b/content/pages/examples/flask/flask-json-jsonify.markdown @@ -72,6 +72,7 @@ def protect(allow_browser_login=False): ## ... source file abbreviated to get to jsonify examples ... + return functools.update_wrapper(wraps, f) def has_access_api(f): @@ -96,7 +97,7 @@ def has_access_api(f): permission_str, self.__class__.__name__ ) ) - response = make_response( +~~ response = make_response( ~~ jsonify( {"message": str(FLAMSG_ERR_SEC_ACCESS_DENIED), "severity": "danger"} ), @@ -164,8 +165,8 @@ from flaskbb.plugins.utils import validate_plugin from flaskbb.user.models import Group, Guest, User from flaskbb.utils.forms import populate_settings_dict, populate_settings_form from flaskbb.utils.helpers import (get_online_users, register_view, - render_template, time_diff, time_utcnow, - FlashAndRedirect) + render_template, redirect_or_next, + time_diff, time_utcnow, FlashAndRedirect) from flaskbb.utils.requirements import (CanBanUser, CanEditUser, IsAdmin, IsAtleastModerator, @@ -173,15 +174,36 @@ from flaskbb.utils.requirements import (CanBanUser, CanEditUser, IsAdmin, ## ... source file abbreviated to get to jsonify examples ... + + flash(_('User updated.'), 'success') + return redirect(url_for('management.edit_user', user_id=user.id)) + + return render_template( + 'management/user_form.html', form=form, title=_('Edit User') + ) + + +class DeleteUser(MethodView): + decorators = [ + allows.requires( + IsAdmin, + on_fail=FlashAndRedirect( + message=_("You are not allowed to manage users"), + level="danger", endpoint="management.overview" ) ) ] def post(self, user_id=None): - if request.is_xhr: - ids = request.get_json()["ids"] - + if request.get_json() is not None: + ids = request.get_json().get("ids") + if not ids: +~~ return jsonify( + message="No ids provided.", + category="error", + status=404 + ) data = [] for user in User.query.filter(User.id.in_(ids)).all(): if current_user.id == user.id: @@ -199,7 +221,7 @@ from flaskbb.utils.requirements import (CanBanUser, CanEditUser, IsAdmin, ) ~~ return jsonify( - message="{} users deleted.".format(len(data)), + message=f"{len(data)} users deleted.", category="success", data=data, status=200 @@ -229,6 +251,36 @@ class AddUser(MethodView): + +class BanUser(MethodView): + decorators = [ + allows.requires( + IsAtleastModerator, + on_fail=FlashAndRedirect( + message=_("You are not allowed to manage users"), + level="danger", + endpoint="management.overview" + ) + ) + ] + + def post(self, user_id=None): + if not Permission(CanBanUser, identity=current_user): + flash( + _("You do not have the permissions to ban this user."), + "danger" + ) + return redirect(url_for("management.overview")) + + if request.get_json() is not None: + ids = request.get_json().get("ids") + if not ids: +~~ return jsonify( + message="No ids provided.", + category="error", + status=404 + ) + data = [] users = User.query.filter(User.id.in_(ids)).all() for user in users: @@ -238,20 +290,13 @@ class AddUser(MethodView): continue elif user.ban(): - data.append( - { - "id": - user.id, - "type": - "ban", - "reverse": - "unban", - "reverse_name": - _("Unban"), - "reverse_url": - url_for("management.unban_user", user_id=user.id) - } - ) + data.append({ + "id": user.id, + "type": "ban", + "reverse": "unban", + "reverse_name": _("Unban"), + "reverse_url": url_for("management.unban_user", user_id=user.id) + }) ~~ return jsonify( message="{} users banned.".format(len(data)), @@ -270,7 +315,8 @@ class AddUser(MethodView): flash(_("User is now banned."), "success") else: flash(_("Could not ban user."), "danger") - return redirect(url_for("management.banned_users")) + + return redirect_or_next(url_for("management.banned_users")) class UnbanUser(MethodView): @@ -278,11 +324,15 @@ class UnbanUser(MethodView): allows.requires( IsAtleastModerator, on_fail=FlashAndRedirect( + message=_("You are not allowed to manage users"), + level="danger", + endpoint="management.overview" + ) + ) + ] -## ... source file abbreviated to get to jsonify examples ... - - + def post(self, user_id=None): if not Permission(CanBanUser, identity=current_user): flash( @@ -291,8 +341,14 @@ class UnbanUser(MethodView): ) return redirect(url_for("management.overview")) - if request.is_xhr: - ids = request.get_json()["ids"] + if request.get_json() is not None: + ids = request.get_json().get("ids") + if not ids: +~~ return jsonify( + message="No ids provided.", + category="error", + status=404 + ) data = [] for user in User.query.filter(User.id.in_(ids)).all(): @@ -300,7 +356,7 @@ class UnbanUser(MethodView): data.append( { "id": user.id, - "type": "unban", + "type": "ban", "reverse": "ban", "reverse_name": _("Ban"), "reverse_url": url_for("management.ban_user", @@ -309,7 +365,7 @@ class UnbanUser(MethodView): ) ~~ return jsonify( - message="{} users unbanned.".format(len(data)), + message=f"{len(data)} users unbanned.", category="success", data=data, status=200 @@ -322,7 +378,7 @@ class UnbanUser(MethodView): else: flash(_("Could not unban user."), "danger") - return redirect(url_for("management.banned_users")) + return redirect_or_next(url_for("management.users")) class Groups(MethodView): @@ -338,6 +394,19 @@ class Groups(MethodView): ## ... source file abbreviated to get to jsonify examples ... + + flash(_('Group updated.'), 'success') + return redirect(url_for('management.groups', group_id=group.id)) + + return render_template( + 'management/group_form.html', form=form, title=_('Edit Group') + ) + + +class DeleteGroup(MethodView): + decorators = [ + allows.requires( + IsAdmin, on_fail=FlashAndRedirect( message=_("You are not allowed to modify groups."), level="danger", @@ -347,8 +416,15 @@ class Groups(MethodView): ] def post(self, group_id=None): - if request.is_xhr: - ids = request.get_json()["ids"] + if request.get_json() is not None: + ids = request.get_json().get("ids") + if not ids: +~~ return jsonify( + message="No ids provided.", + category="error", + status=404 + ) + if not (set(ids) & set(["1", "2", "3", "4", "5", "6"])): data = [] for group in Group.query.filter(Group.id.in_(ids)).all(): @@ -399,6 +475,21 @@ class Groups(MethodView): ## ... source file abbreviated to get to jsonify examples ... + reports = Report.query.\ + filter(Report.zapped == None).\ + order_by(Report.id.desc()).\ + paginate(page, flaskbb_config['USERS_PER_PAGE'], False) + + return render_template("management/reports.html", reports=reports) + + +class MarkReportRead(MethodView): + decorators = [ + allows.requires( + IsAtleastModerator, + on_fail=FlashAndRedirect( + message=_("You are not allowed to view reports."), + level="danger", endpoint="management.overview" ) ) @@ -406,8 +497,14 @@ class Groups(MethodView): def post(self, report_id=None): - if request.is_xhr: - ids = request.get_json()["ids"] + if request.get_json() is not None: + ids = request.get_json().get("ids") + if not ids: +~~ return jsonify( + message="No ids provided.", + category="error", + status=404 + ) data = [] for report in Report.query.filter(Report.id.in_(ids)).all(): @@ -438,13 +535,13 @@ class Groups(MethodView): _("Report %(id)s is already marked as read.", id=report.id), "success" ) - return redirect(url_for("management.reports")) + return redirect_or_next(url_for("management.reports")) report.zapped_by = current_user.id report.zapped = time_utcnow() report.save() flash(_("Report %(id)s marked as read.", id=report.id), "success") - return redirect(url_for("management.reports")) + return redirect_or_next(url_for("management.reports")) reports = Report.query.filter(Report.zapped == None).all() report_list = [] @@ -454,6 +551,20 @@ class Groups(MethodView): ## ... source file abbreviated to get to jsonify examples ... + report_list.append(report) + + db.session.add_all(report_list) + db.session.commit() + + flash(_("All reports were marked as read."), "success") + return redirect_or_next(url_for("management.reports")) + + +class DeleteReport(MethodView): + decorators = [ + allows.requires( + IsAtleastModerator, + on_fail=FlashAndRedirect( message=_("You are not allowed to view reports."), level="danger", endpoint="management.overview" @@ -462,11 +573,16 @@ class Groups(MethodView): ] def post(self, report_id=None): + if request.get_json() is not None: + ids = request.get_json().get("ids") + if not ids: +~~ return jsonify( + message="No ids provided.", + category="error", + status=404 + ) - if request.is_xhr: - ids = request.get_json()["ids"] data = [] - for report in Report.query.filter(Report.id.in_(ids)).all(): if report.delete(): data.append( @@ -489,7 +605,7 @@ class Groups(MethodView): report = Report.query.filter_by(id=report_id).first_or_404() report.delete() flash(_("Report deleted."), "success") - return redirect(url_for("management.reports")) + return redirect_or_next(url_for("management.reports")) class CeleryStatus(MethodView): @@ -543,12 +659,121 @@ class ManagementOverview(MethodView): ``` -## Example 3 from flaskSaaS +## Example 3 from Flask-Meld +[Flask-Meld](https://github.com/mikeabrahamsen/Flask-Meld) +([PyPI package information](https://pypi.org/project/Flask-Meld/)) +allows you to write your front end web code in your back end +Python code. It does this by adding a `{% meld_scripts %}` tag to +the Flask template engine and then inserting components written +in Python scripts created by a developer. + +[**Flask-Meld / flask_meld / component.py**](https://github.com/mikeabrahamsen/Flask-Meld/blob/main/flask_meld/./component.py) + +```python +# component.py +import os +import uuid +from importlib.util import module_from_spec, spec_from_file_location +from itertools import groupby +from operator import itemgetter + +import orjson +from bs4 import BeautifulSoup +from bs4.element import Tag +from bs4.formatter import HTMLFormatter +~~from flask import current_app, jsonify, render_template +from jinja2.exceptions import TemplateNotFound + + +def convert_to_snake_case(s): + s.replace("-", "_") + return s + + +def convert_to_camel_case(s): + s = convert_to_snake_case(s) + return "".join(word.title() for word in s.split("_")) + + +def get_component_class(component_name): + module_name = convert_to_snake_case(component_name) + class_name = convert_to_camel_case(module_name) + module = get_component_module(module_name) + component_class = getattr(module, class_name) + + return component_class + + +def get_component_module(module_name): + user_specified_dir = current_app.config.get("MELD_COMPONENT_DIR", None) + + +## ... source file abbreviated to get to jsonify examples ... + + + + def _render_template(self, template_name: str, context_variables: dict): + try: + return render_template(template_name, **context_variables) + except TemplateNotFound: + return render_template(f"meld/{template_name}", **context_variables) + + def _view(self, component_name: str): + data = self._attributes() + context = self.__context__() + context_variables = {} + context_variables.update(context["attributes"]) + context_variables.update(context["methods"]) + context_variables.update({"form": self._form}) + + rendered_template = self._render_template( + f"{component_name}.html", context_variables + ) + + soup = BeautifulSoup(rendered_template, features="html.parser") + root_element = Component._get_root_element(soup) + root_element["meld:id"] = str(self.id) + self._set_values(root_element, context_variables) + + script = soup.new_tag("script", type="module") +~~ init = {"id": str(self.id), "name": component_name, "data": jsonify(data).json} + init_json = orjson.dumps(init).decode("utf-8") + + meld_import = 'import {Meld} from "/meld_js_src/meld.js";' + script.string = f"{meld_import} Meld.componentInit({init_json});" + root_element.append(script) + + rendered_template = Component._desoupify(soup) + + return rendered_template + + def _set_values(self, soup, context_variables): + for element in soup.select("input,select,textarea"): + model_attrs = [ + attr for attr in element.attrs.keys() if attr.startswith("meld:model") + ] + if len(model_attrs) > 1: + raise Exception( + "Multiple 'meld:model' attributes not allowed on one tag." + ) + + for model_attr in model_attrs: + value = context_variables[element.attrs[model_attr]] + element.attrs["value"] = value + if element.name == "select": + + +## ... source file continues with no further jsonify examples... + +``` + + +## Example 4 from flaskSaaS [flaskSaas](https://github.com/alectrocute/flaskSaaS) is a boilerplate starter project to build a software-as-a-service (SaaS) web application in [Flask](/flask.html), with [Stripe](/stripe.html) for billing. The boilerplate relies on many common Flask extensions such as -[Flask-WTF](https://flask-wtf.readthedocs.io/en/latest/), +[Flask-WTF](https://flask-wtf.readthedocs.io/), [Flask-Login](https://flask-login.readthedocs.io/en/latest/), [Flask-Admin](https://flask-admin.readthedocs.io/en/latest/), and many others. The project is provided as open source under the @@ -593,7 +818,7 @@ def contact(): ``` -## Example 4 from Flask-SocketIO +## Example 5 from Flask-SocketIO [Flask-SocketIO](https://github.com/miguelgrinberg/Flask-SocketIO) ([PyPI package information](https://pypi.org/project/Flask-SocketIO/), [official tutorial](https://blog.miguelgrinberg.com/post/easy-websockets-with-flask-and-gevent) @@ -675,7 +900,7 @@ def get_session(): ``` -## Example 5 from Datadog Flask Example App +## Example 6 from Datadog Flask Example App The [Datadog Flask example app](https://github.com/DataDog/trace-examples/tree/master/python/flask) contains many examples of the [Flask](/flask.html) core functions available to a developer using the [web framework](/web-frameworks.html). @@ -777,7 +1002,7 @@ def stream(): ``` -## Example 6 from indico +## Example 7 from indico [indico](https://github.com/indico/indico) ([project website](https://getindico.io/), [documentation](https://docs.getindico.io/en/stable/installation/) @@ -791,16 +1016,19 @@ The code is open sourced under the ```python # util.py -from __future__ import absolute_import, unicode_literals - +import hashlib +import sys from datetime import datetime -~~from flask import g, has_request_context, jsonify, render_template, request, session +import sentry_sdk +from authlib.oauth2 import OAuth2Error +~~from flask import flash, g, has_request_context, jsonify, render_template, request, session from itsdangerous import Signer from markupsafe import Markup -from werkzeug.exceptions import ImATeapot +from werkzeug.exceptions import BadRequest, Forbidden, ImATeapot from werkzeug.urls import url_decode, url_encode, url_parse, url_unparse +from indico.util.caching import memoize_request from indico.util.i18n import _ from indico.web.flask.templating import get_template_module @@ -821,7 +1049,7 @@ def _pop_injected_js(): def jsonify_form(form, fields=None, submit=None, back=None, back_url=None, back_button=True, disabled_until_change=True, disabled_fields=(), form_header_kwargs=None, skip_labels=False, save_reminder=False, - footer_align_right=False, disable_if_locked=True): + footer_align_right=False, disable_if_locked=True, message=None): if submit is None: submit = _('Save') if back is None: @@ -832,7 +1060,7 @@ def jsonify_form(form, fields=None, submit=None, back=None, back_url=None, back_ html = tpl.simple_form(form, fields=fields, submit=submit, back=back, back_url=back_url, back_button=back_button, disabled_until_change=disabled_until_change, disabled_fields=disabled_fields, form_header_kwargs=form_header_kwargs, skip_labels=skip_labels, save_reminder=save_reminder, - footer_align_right=footer_align_right, disable_if_locked=disable_if_locked) + footer_align_right=footer_align_right, disable_if_locked=disable_if_locked, message=message) ~~ return jsonify(html=html, js=_pop_injected_js()) @@ -853,19 +1081,19 @@ def jsonify_data(flash=True, **json_data): class ExpectedError(ImATeapot): def __init__(self, message, **data): - super(ExpectedError, self).__init__(message or 'Something went wrong') + super().__init__(message or 'Something went wrong') self.data = dict(data, message=message) def _format_request_data(data, hide_passwords=False): - if not hasattr(data, 'iterlists'): - data = ((k, [v]) for k, v in data.iteritems()) + if not hasattr(data, 'lists'): + data = ((k, [v]) for k, v in data.items()) else: - data = data.iterlists() + data = data.lists() rv = {} for key, values in data: if hide_passwords and 'password' in key: - values = [v if not v else '<{} chars hidden>'.format(len(v)) for v in values] + values = [v if not v else f'<{len(v)} chars hidden>' for v in values] rv[key] = values if len(values) != 1 else values[0] return rv @@ -880,7 +1108,7 @@ def get_request_info(hide_passwords=True): ``` -## Example 7 from keras-flask-deploy-webapp +## Example 8 from keras-flask-deploy-webapp The [keras-flask-deploy-webapp](https://github.com/mtobeiyf/keras-flask-deploy-webapp) project combines the [Flask](/flask.html) [web framework](/web-frameworks.html) @@ -915,13 +1143,13 @@ from util import base64_to_pil app = Flask(__name__) -from keras.applications.mobilenet_v2 import MobileNetV2 + +from tensorflow.keras.applications.mobilenet_v2 import MobileNetV2 model = MobileNetV2(weights='imagenet') print('Model loaded. Check http://127.0.0.1:5000/') -MODEL_PATH = 'models/your_model.h5' ## ... source file abbreviated to get to jsonify examples ... @@ -969,7 +1197,7 @@ if __name__ == '__main__': ``` -## Example 8 from sandman2 +## Example 9 from sandman2 [sandman2](https://github.com/jeffknupp/sandman2) ([project documentation](https://sandman2.readthedocs.io/en/latest/) and @@ -1137,7 +1365,7 @@ class Service(MethodView): ``` -## Example 9 from tedivms-flask +## Example 10 from tedivms-flask [tedivm's flask starter app](https://github.com/tedivm/tedivms-flask) is a base of [Flask](/flask.html) code and related projects such as [Celery](/celery.html) which provides a template to start your own diff --git a/content/pages/examples/flask/flask-sessions-badsignature.markdown b/content/pages/examples/flask/flask-sessions-badsignature.markdown index 4428d6b6e..afe01dee0 100644 --- a/content/pages/examples/flask/flask-sessions-badsignature.markdown +++ b/content/pages/examples/flask/flask-sessions-badsignature.markdown @@ -18,67 +18,7 @@ and SessionMixin are a couple of other callables within the `flask.sessions` package that also have code examples. -## Example 1 from FlaskBB -[FlaskBB](https://github.com/flaskbb/flaskbb) -([project website](https://flaskbb.org/)) is a [Flask](/flask.html)-based -forum web application. The web app allows users to chat in an open -message board or send private messages in plain text or -[Markdown](/markdown.html). - -FlaskBB is provided as open source -[under this license](https://github.com/flaskbb/flaskbb/blob/master/LICENSE). - -[**FlaskBB / flaskbb / tokens / serializer.py**](https://github.com/flaskbb/flaskbb/blob/master/flaskbb/tokens/serializer.py) - -```python -# serializer.py - -from datetime import timedelta - -~~from itsdangerous import (BadData, BadSignature, SignatureExpired, - TimedJSONWebSignatureSerializer) - -from ..core import tokens - - -_DEFAULT_EXPIRY = timedelta(hours=1) - - -class FlaskBBTokenSerializer(tokens.TokenSerializer): - - def __init__(self, secret_key, expiry=_DEFAULT_EXPIRY): - self._serializer = TimedJSONWebSignatureSerializer( - secret_key, int(expiry.total_seconds()) - ) - - def dumps(self, token): - return self._serializer.dumps( - { - 'id': token.user_id, - 'op': token.operation, - } - ) - - def loads(self, raw_token): - try: - parsed = self._serializer.loads(raw_token) - except SignatureExpired: - raise tokens.TokenError.expired() -~~ except BadSignature: # pragma: no branch - raise tokens.TokenError.invalid() - except BadData: # pragma: no cover - raise tokens.TokenError.bad() - else: - return tokens.Token(user_id=parsed['id'], operation=parsed['op']) - - - -## ... source file continues with no further BadSignature examples... - -``` - - -## Example 2 from flask-base +## Example 1 from flask-base [flask-base](https://github.com/hack4impact/flask-base) ([project documentation](http://hack4impact.github.io/flask-base/)) provides boilerplate code for new [Flask](/flask.html) web apps. @@ -88,7 +28,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) backend, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling and many others. flask-base is provided as open source under the @@ -219,7 +159,7 @@ class Role(db.Model): ``` -## Example 3 from flask-bones +## Example 2 from flask-bones [flask-bones](https://github.com/cburmeister/flask-bones) ([demo](http://flask-bones.herokuapp.com/)) is large scale [Flask](/flask.html) example application built @@ -319,7 +259,7 @@ def verify(token): ``` -## Example 4 from Flask-Security-Too +## Example 3 from Flask-Security-Too [Flask-Security-Too](https://github.com/Flask-Middleware/flask-security/) ([PyPi page](https://pypi.org/project/Flask-Security-Too/) and [project documentation](https://flask-security-too.readthedocs.io/en/stable/)) @@ -338,21 +278,23 @@ The Flask-Security-Too project is provided as open source under the ```python # utils.py -import abc -import base64 -import datetime -from functools import partial -import hashlib -import hmac -import time -from typing import Dict, List +import typing as t import warnings -from datetime import timedelta from urllib.parse import parse_qsl, parse_qs, urlsplit, urlunsplit, urlencode import urllib.request import urllib.error -from flask import _request_ctx_stack, current_app, flash, g, request, session, url_for +from flask import ( + _request_ctx_stack, + after_this_request, + current_app, + flash, + g, + request, + render_template, + session, + url_for, +) from flask.json import JSONEncoder from flask_login import login_user as _login_user from flask_login import logout_user as _logout_user @@ -360,21 +302,22 @@ from flask_login import current_user from flask_login import COOKIE_NAME as REMEMBER_COOKIE_NAME from flask_principal import AnonymousIdentity, Identity, identity_changed, Need from flask_wtf import csrf -from wtforms import validators, ValidationError +from wtforms import ValidationError ~~from itsdangerous import BadSignature, SignatureExpired -from speaklater import is_lazy_string +from werkzeug import __version__ as werkzeug_version from werkzeug.local import LocalProxy from werkzeug.datastructures import MultiDict -from .quart_compat import best -from .signals import user_authenticated -_security = LocalProxy(lambda: current_app.extensions["security"]) +from .quart_compat import best, get_quart_status +from .proxies import _security, _datastore, _pwd_context, _hashing_context +from .signals import user_authenticated -_datastore = LocalProxy(lambda: _security.datastore) +if t.TYPE_CHECKING: # pragma: no cover + from flask import Flask, Response + from .datastore import User -_pwd_context = LocalProxy(lambda: _security.pwd_context) +SB = t.Union[str, bytes] -_hashing_context = LocalProxy(lambda: _security.hashing_context) localize_callback = LocalProxy(lambda: _security.i18n_domain.gettext) @@ -387,21 +330,20 @@ def _(translate): - ## ... source file abbreviated to get to BadSignature examples ... - if config_value("EMAIL_PLAINTEXT"): - body = _security.render_template("%s/%s.txt" % ctx, **context) - if config_value("EMAIL_HTML"): - html = _security.render_template("%s/%s.html" % ctx, **context) - sender = _security.email_sender if isinstance(sender, LocalProxy): sender = sender._get_current_object() + if isinstance(sender, tuple) and len(sender) == 2: + sender = (str(sender[0]), str(sender[1])) + else: + sender = str(sender) + _security._mail_util.send_mail( - template, subject, recipient, str(sender), body, html, context.get("user", None) + template, subject, recipient, sender, body, html, context.get("user", None) ) @@ -430,8 +372,10 @@ def get_token_status(token, serializer, max_age=None, return_data=False): return expired, invalid, user -def check_and_get_token_status(token, serializer, within=None): - serializer = getattr(_security, serializer + "_serializer") +def check_and_get_token_status( + token: str, serializer_name: str, within: datetime.timedelta +) -> t.Tuple[bool, bool, t.Any]: + serializer = getattr(_security, serializer_name + "_serializer") max_age = within.total_seconds() data = None expired, invalid = False, False @@ -447,7 +391,7 @@ def check_and_get_token_status(token, serializer, within=None): return expired, invalid, data -def get_identity_attributes(app=None) -> List: +def get_identity_attributes(app: t.Optional["Flask"] = None) -> t.List[str]: app = app or current_app iattrs = app.config["SECURITY_USER_IDENTITY_ATTRIBUTES"] if iattrs: @@ -455,7 +399,9 @@ def get_identity_attributes(app=None) -> List: return [] -def get_identity_attribute(attr, app=None) -> Dict: +def get_identity_attribute( + attr: str, app: t.Optional["Flask"] = None +) -> t.Dict[str, t.Any]: app = app or current_app iattrs = app.config["SECURITY_USER_IDENTITY_ATTRIBUTES"] if iattrs: @@ -464,8 +410,6 @@ def get_identity_attribute(attr, app=None) -> Dict: ] if details: return details[0] - return {} - ## ... source file continues with no further BadSignature examples... diff --git a/content/pages/examples/flask/flask-sessions-sessioninterface.markdown b/content/pages/examples/flask/flask-sessions-sessioninterface.markdown index 757772a3d..ad230c936 100644 --- a/content/pages/examples/flask/flask-sessions-sessioninterface.markdown +++ b/content/pages/examples/flask/flask-sessions-sessioninterface.markdown @@ -40,7 +40,8 @@ import os import requests import yaml -from flask import Flask, session, render_template +from flask import Flask, render_template +from flask import session as current_session from flask_mail import Mail from flask_migrate import Migrate, MigrateCommand ~~from flask.sessions import SessionInterface @@ -111,7 +112,7 @@ def init_session_manager(app): @user_logged_out.connect_via(app) def clear_session(sender, user, **extra): - session.clear() + current_session.clear() def init_celery_service(app): diff --git a/content/pages/examples/flask/flask-sessions-sessionmixin.markdown b/content/pages/examples/flask/flask-sessions-sessionmixin.markdown index 57ac14477..87a1b97b1 100644 --- a/content/pages/examples/flask/flask-sessions-sessionmixin.markdown +++ b/content/pages/examples/flask/flask-sessions-sessionmixin.markdown @@ -43,7 +43,7 @@ import sys gevent_socketio_found = True try: - from socketio import socketio_manage + from socketio import socketio_manage # noqa: F401 except ImportError: gevent_socketio_found = False if gevent_socketio_found: @@ -56,14 +56,14 @@ import flask from flask import _request_ctx_stack, has_request_context, json as flask_json ~~from flask.sessions import SessionMixin import socketio -from socketio.exceptions import ConnectionRefusedError +from socketio.exceptions import ConnectionRefusedError # noqa: F401 from werkzeug.debug import DebuggedApplication from werkzeug.serving import run_with_reloader from .namespace import Namespace from .test_client import SocketIOTestClient -__version__ = '4.3.2dev' +__version__ = '5.0.2dev' class _SocketIOMiddleware(socketio.WSGIApp): diff --git a/content/pages/examples/flask/flask-signals-got-request-exception.markdown b/content/pages/examples/flask/flask-signals-got-request-exception.markdown index 4484cce5a..dc2a466c4 100644 --- a/content/pages/examples/flask/flask-signals-got-request-exception.markdown +++ b/content/pages/examples/flask/flask-signals-got-request-exception.markdown @@ -41,6 +41,7 @@ import operator import re import six import sys +import warnings from collections import OrderedDict from functools import wraps, partial @@ -48,7 +49,10 @@ from types import MethodType from flask import url_for, request, current_app from flask import make_response as original_flask_make_response -from flask.helpers import _endpoint_from_view_func +try: + from flask.helpers import _endpoint_from_view_func +except ImportError: + from flask.scaffold import _endpoint_from_view_func ~~from flask.signals import got_request_exception from jsonschema import RefResolver @@ -62,54 +66,25 @@ from werkzeug.exceptions import ( NotAcceptable, InternalServerError, ) -from werkzeug.wrappers import BaseResponse + +from werkzeug import __version__ as werkzeug_version + +if werkzeug_version.split('.')[0] >= '2': + from werkzeug.wrappers import Response as BaseResponse +else: + from werkzeug.wrappers import BaseResponse from . import apidoc from .mask import ParseError, MaskError from .namespace import Namespace from .postman import PostmanCollectionV1 -from .resource import Resource -from .swagger import Swagger -from .utils import default_id, camel_to_dash, unpack -from .representations import output_json -from ._http import HTTPStatus - ## ... source file abbreviated to get to got_request_exception examples ... - except MethodNotAllowed as e: - valid_route_method = e.valid_methods[0] - rule, _ = adapter.match(method=valid_route_method, return_rule=True) - return self.owns_endpoint(rule.endpoint) - except NotFound: - return self.catch_all_404s - except Exception: - pass - - def _has_fr_route(self): - if self._should_use_fr_error_handler(): - return True - if not request.url_rule: - return False - return self.owns_endpoint(request.url_rule.endpoint) - - def error_router(self, original_handler, e): - if self._has_fr_route(): - try: - return self.handle_error(e) - except Exception as f: - return original_handler(f) - return original_handler(e) - - def handle_error(self, e): -~~ got_request_exception.send(current_app._get_current_object(), exception=e) - - if ( - not isinstance(e, HTTPException) and current_app.propagate_exceptions - and not isinstance(e, tuple(self.error_handlers.keys())) + and not isinstance(e, tuple(self._own_and_child_error_handlers.keys())) ): exc_type, exc_value, tb = sys.exc_info() @@ -129,6 +104,35 @@ from ._http import HTTPStatus if isinstance(e, typecheck): result = handler(e) default_data, code, headers = unpack( + result, HTTPStatus.INTERNAL_SERVER_ERROR + ) + break + else: +~~ got_request_exception.send(current_app._get_current_object(), exception=e) + + if isinstance(e, HTTPException): + code = HTTPStatus(e.code) + if include_message_in_response: + default_data = {"message": getattr(e, "description", code.phrase)} + headers = e.get_response().headers + elif self._default_error_handler: + result = self._default_error_handler(e) + default_data, code, headers = unpack( + result, HTTPStatus.INTERNAL_SERVER_ERROR + ) + else: + code = HTTPStatus.INTERNAL_SERVER_ERROR + if include_message_in_response: + default_data = { + "message": code.phrase, + } + + if include_message_in_response: + default_data["message"] = default_data.get("message", str(e)) + + data = getattr(e, "data", default_data) + fallback_mediatype = None + ## ... source file continues with no further got_request_exception examples... diff --git a/content/pages/examples/flask/flask-signals-namespace.markdown b/content/pages/examples/flask/flask-signals-namespace.markdown index 4faeecf05..f9987d40e 100644 --- a/content/pages/examples/flask/flask-signals-namespace.markdown +++ b/content/pages/examples/flask/flask-signals-namespace.markdown @@ -108,41 +108,51 @@ from sqlalchemy import event from sqlalchemy import inspect from sqlalchemy import orm from sqlalchemy.engine.url import make_url -from sqlalchemy.ext.declarative import declarative_base -from sqlalchemy.ext.declarative import DeclarativeMeta from sqlalchemy.orm.exc import UnmappedClassError from sqlalchemy.orm.session import Session as SessionBase from .model import DefaultMeta from .model import Model -__version__ = "3.0.0.dev" +try: + from sqlalchemy.orm import declarative_base + from sqlalchemy.orm import DeclarativeMeta +except ImportError: + from sqlalchemy.ext.declarative import declarative_base + from sqlalchemy.ext.declarative import DeclarativeMeta + +try: + from greenlet import getcurrent as _ident_func +except ImportError: + from threading import get_ident as _ident_func + +__version__ = "3.0.0.dev0" ~~_signals = Namespace() models_committed = _signals.signal("models-committed") before_models_committed = _signals.signal("before-models-committed") -def _make_table(db): - def _make_table(*args, **kwargs): - if len(args) > 1 and isinstance(args[1], db.Column): - args = (args[0], db.metadata) + args[1:] - info = kwargs.pop("info", None) or {} - info.setdefault("bind_key", None) - kwargs["info"] = info - return sqlalchemy.Table(*args, **kwargs) +def _sa_url_set(url, **kwargs): + try: + url = url.set(**kwargs) + except AttributeError: + for key, value in kwargs.items(): + setattr(url, key, value) - return _make_table + return url -def _set_default_query_class(d, cls): - if "query_class" not in d: - d["query_class"] = cls +def _sa_url_query_setdefault(url, **kwargs): + query = dict(url.query) + for key, value in kwargs.items(): + query.setdefault(key, value) -def _wrap_with_default_query_class(fn, cls): - @functools.wraps(fn) - def newfn(*args, **kwargs): + return _sa_url_set(url, query=query) + + +def _make_table(db): ## ... source file continues with no further Namespace examples... diff --git a/content/pages/examples/flask/flask-templating-render-template-string.markdown b/content/pages/examples/flask/flask-templating-render-template-string.markdown index c785805cb..7f74cb079 100644 --- a/content/pages/examples/flask/flask-templating-render-template-string.markdown +++ b/content/pages/examples/flask/flask-templating-render-template-string.markdown @@ -20,14 +20,119 @@ the `.templating` part. render_template is another callable from the `flask.templating` package with code examples. -These topics are also useful while reading the `render_template_string` examples: +These subjects go along with the `render_template_string` code examples: * [template engines](/template-engines.html), specifically [Jinja2](/jinja2.html) * [Flask](/flask.html) and the concepts for [web frameworks](/web-frameworks.html) * [Cascading Style Sheets (CSS)](/cascading-style-sheets.html) and [web design](/web-design.html) -## Example 1 from Flask-User +## Example 1 from CTFd +[CTFd](https://github.com/CTFd/CTFd) +([homepage](https://ctfd.io/)) is a +[capture the flag (CTF) hacking web app](https://cybersecurity.att.com/blogs/security-essentials/capture-the-flag-ctf-what-is-it-for-a-newbie) +built with [Flask](/flask.html). The application can be used +as-is to run CTF events, or modified for custom rules for related +scenarios. CTFd is open sourced under the +[Apache License 2.0](https://github.com/CTFd/CTFd/blob/master/LICENSE). + +[**CTFd / tests / test_themes.py**](https://github.com/CTFd/CTFd/blob/master/./tests/test_themes.py) + +```python +# test_themes.py + +import os +import shutil + +import pytest +~~from flask import render_template, render_template_string, request +from jinja2.exceptions import TemplateNotFound +from jinja2.sandbox import SecurityError +from werkzeug.test import Client + +from CTFd.config import TestingConfig +from CTFd.utils import get_config, set_config +from tests.helpers import create_ctfd, destroy_ctfd, gen_user, login_as_user + + +def test_themes_run_in_sandbox(): + app = create_ctfd() + with app.app_context(): + try: + app.jinja_env.from_string( + "{{ ().__class__.__bases__[0].__subclasses__()[40]('./test_utils.py').read() }}" + ).render() + except SecurityError: + pass + except Exception as e: + raise e + destroy_ctfd(app) + + +def test_themes_cant_access_configpy_attributes(): + + +## ... source file abbreviated to get to render_template_string examples ... + + + except TemplateNotFound: + pass + try: + r = client.get("/themes/foo_fallback/static/js/pages/main.dev.js") + except TemplateNotFound: + pass + destroy_ctfd(app) + + app = create_ctfd() + with app.app_context(): + set_config("ctf_theme", "foo_fallback") + assert app.config["THEME_FALLBACK"] == True + with app.test_client() as client: + r = client.get("/") + assert r.status_code == 200 + r = client.get("/themes/foo_fallback/static/js/pages/main.dev.js") + assert r.status_code == 200 + destroy_ctfd(app) + + os.rmdir(os.path.join(app.root_path, "themes", "foo_fallback")) + + +def test_theme_template_loading_by_prefix(): + app = create_ctfd() + with app.test_request_context(): +~~ tpl1 = render_template_string("{% extends 'core/page.html' %}", content="test") + tpl2 = render_template("page.html", content="test") + assert tpl1 == tpl2 + + +def test_theme_template_disallow_loading_admin_templates(): + app = create_ctfd() + with app.app_context(): + try: + filename = os.path.join( + app.root_path, "themes", "foo_disallow", "admin", "malicious.html" + ) + os.makedirs(os.path.dirname(filename), exist_ok=True) + set_config("ctf_theme", "foo_disallow") + with open(filename, "w") as f: + f.write("malicious") + + with pytest.raises(TemplateNotFound): +~~ render_template_string("{% include 'admin/malicious.html' %}") + finally: + shutil.rmtree( + os.path.join(app.root_path, "themes", "foo_disallow"), + ignore_errors=True, + ) + + + +## ... source file continues with no further render_template_string examples... + +``` + + +## Example 2 from Flask-User [Flask-User](https://github.com/lingthio/Flask-User) ([PyPI information](https://pypi.org/project/Flask-User/) and @@ -130,7 +235,7 @@ class ConfigClass(object): ``` -## Example 2 from Datadog Flask Example App +## Example 3 from Datadog Flask Example App The [Datadog Flask example app](https://github.com/DataDog/trace-examples/tree/master/python/flask) contains many examples of the [Flask](/flask.html) core functions available to a developer using the [web framework](/web-frameworks.html). diff --git a/content/pages/examples/flask/flask-templating-render-template.markdown b/content/pages/examples/flask/flask-templating-render-template.markdown index da6a6da1a..9d320fab4 100644 --- a/content/pages/examples/flask/flask-templating-render-template.markdown +++ b/content/pages/examples/flask/flask-templating-render-template.markdown @@ -21,7 +21,7 @@ the `.templating` part. render_template_string is another callable from the `flask.templating` package with code examples. -You should read up on these subjects along with these `render_template` examples: +These topics are also useful while reading the `render_template` examples: * [template engines](/template-engines.html), specifically [Jinja2](/jinja2.html) * [Flask](/flask.html) and the concepts for [web frameworks](/web-frameworks.html) @@ -128,353 +128,94 @@ as-is to run CTF events, or modified for custom rules for related scenarios. CTFd is open sourced under the [Apache License 2.0](https://github.com/CTFd/CTFd/blob/master/LICENSE). -[**CTFd / CTFd / auth.py**](https://github.com/CTFd/CTFd/blob/master/./CTFd/auth.py) +[**CTFd / tests / test_themes.py**](https://github.com/CTFd/CTFd/blob/master/./tests/test_themes.py) ```python -# auth.py -import base64 +# test_themes.py -import requests -from flask import Blueprint -from flask import current_app as app -~~from flask import redirect, render_template, request, session, url_for -from itsdangerous.exc import BadSignature, BadTimeSignature, SignatureExpired - -from CTFd.cache import clear_team_session, clear_user_session -from CTFd.models import Teams, Users, db -from CTFd.utils import config, email, get_app_config, get_config -from CTFd.utils import user as current_user -from CTFd.utils import validators -from CTFd.utils.config import is_teams_mode -from CTFd.utils.config.integrations import mlc_registration -from CTFd.utils.config.visibility import registration_visible -from CTFd.utils.crypto import verify_password -from CTFd.utils.decorators import ratelimit -from CTFd.utils.decorators.visibility import check_registration_visibility -from CTFd.utils.helpers import error_for, get_errors, markup -from CTFd.utils.logging import log -from CTFd.utils.modes import TEAMS_MODE -from CTFd.utils.security.auth import login_user, logout_user -from CTFd.utils.security.signing import unserialize -from CTFd.utils.validators import ValidationError - -auth = Blueprint("auth", __name__) - - -@auth.route("/confirm", methods=["POST", "GET"]) -@auth.route("/confirm/", methods=["POST", "GET"]) -@ratelimit(method="POST", limit=10, interval=60) -def confirm(data=None): - if not get_config("verify_emails"): - return redirect(url_for("challenges.listing")) - - if data and request.method == "GET": - try: - user_email = unserialize(data, max_age=1800) - except (BadTimeSignature, SignatureExpired): -~~ return render_template( - "confirm.html", errors=["Your confirmation link has expired"] - ) - except (BadSignature, TypeError, base64.binascii.Error): -~~ return render_template( - "confirm.html", errors=["Your confirmation token is invalid"] - ) +import os +import shutil - user = Users.query.filter_by(email=user_email).first_or_404() - if user.verified: - return redirect(url_for("views.settings")) +import pytest +~~from flask import render_template, render_template_string, request +from jinja2.exceptions import TemplateNotFound +from jinja2.sandbox import SecurityError +from werkzeug.test import Client - user.verified = True - log( - "registrations", - format="[{date}] {ip} - successful confirmation for {name}", - name=user.name, - ) - db.session.commit() - clear_user_session(user_id=user.id) - email.successful_registration_notification(user.email) - db.session.close() - if current_user.authed(): - return redirect(url_for("challenges.listing")) - return redirect(url_for("auth.login")) - - if current_user.authed() is False: - return redirect(url_for("auth.login")) - - user = Users.query.filter_by(id=session["id"]).first_or_404() - if user.verified: - return redirect(url_for("views.settings")) - - if data is None: - if request.method == "POST": - email.verify_email_address(user.email) - log( - "registrations", - format="[{date}] {ip} - {name} initiated a confirmation email resend", - ) -~~ return render_template( - "confirm.html", infos=[f"Confirmation email sent to {user.email}!"] - ) - elif request.method == "GET": -~~ return render_template("confirm.html") - - -@auth.route("/reset_password", methods=["POST", "GET"]) -@auth.route("/reset_password/", methods=["POST", "GET"]) -@ratelimit(method="POST", limit=10, interval=60) -def reset_password(data=None): - if config.can_send_mail() is False: -~~ return render_template( - "reset_password.html", - errors=[ - markup( - "This CTF is not configured to send email.
Please contact an organizer to have your password reset." - ) - ], - ) +from CTFd.config import TestingConfig +from CTFd.utils import get_config, set_config +from tests.helpers import create_ctfd, destroy_ctfd, gen_user, login_as_user - if data is not None: - try: - email_address = unserialize(data, max_age=1800) - except (BadTimeSignature, SignatureExpired): -~~ return render_template( - "reset_password.html", errors=["Your link has expired"] - ) - except (BadSignature, TypeError, base64.binascii.Error): -~~ return render_template( - "reset_password.html", errors=["Your reset token is invalid"] - ) - if request.method == "GET": -~~ return render_template("reset_password.html", mode="set") - if request.method == "POST": - password = request.form.get("password", "").strip() - user = Users.query.filter_by(email=email_address).first_or_404() - if user.oauth_id: -~~ return render_template( - "reset_password.html", - infos=[ - "Your account was registered via an authentication provider and does not have an associated password. Please login via your authentication provider." - ], - ) - - pass_short = len(password) == 0 - if pass_short: -~~ return render_template( - "reset_password.html", errors=["Please pick a longer password"] - ) - - user.password = password - db.session.commit() - clear_user_session(user_id=user.id) - log( - "logins", - format="[{date}] {ip} - successful password reset for {name}", - name=user.name, - ) - db.session.close() - email.password_change_alert(user.email) - return redirect(url_for("auth.login")) - - if request.method == "POST": - email_address = request.form["email"].strip() - user = Users.query.filter_by(email=email_address).first() - - get_errors() - - if not user: -~~ return render_template( - "reset_password.html", - infos=[ - "If that account exists you will receive an email, please check your inbox" - ], - ) +def test_themes_run_in_sandbox(): + app = create_ctfd() + with app.app_context(): + try: + app.jinja_env.from_string( + "{{ ().__class__.__bases__[0].__subclasses__()[40]('./test_utils.py').read() }}" + ).render() + except SecurityError: + pass + except Exception as e: + raise e + destroy_ctfd(app) - if user.oauth_id: -~~ return render_template( - "reset_password.html", - infos=[ - "The email address associated with this account was registered via an authentication provider and does not have an associated password. Please login via your authentication provider." - ], - ) - email.forgot_password(email_address) - -~~ return render_template( - "reset_password.html", - infos=[ - "If that account exists you will receive an email, please check your inbox" - ], - ) -~~ return render_template("reset_password.html") - - -@auth.route("/register", methods=["POST", "GET"]) -@check_registration_visibility -@ratelimit(method="POST", limit=10, interval=5) -def register(): - errors = get_errors() - if request.method == "POST": - name = request.form.get("name", "").strip() - email_address = request.form.get("email", "").strip().lower() - password = request.form.get("password", "").strip() - - website = request.form.get("website") - affiliation = request.form.get("affiliation") - country = request.form.get("country") - - name_len = len(name) == 0 - names = Users.query.add_columns("name", "id").filter_by(name=name).first() - emails = ( - Users.query.add_columns("email", "id") - .filter_by(email=email_address) - .first() - ) - pass_short = len(password) == 0 +def test_themes_cant_access_configpy_attributes(): ## ... source file abbreviated to get to render_template examples ... - errors.append( - "Only email addresses under {domains} may register".format( - domains=get_config("domain_whitelist") - ) - ) - if names: - errors.append("That user name is already taken") - if team_name_email_check is True: - errors.append("Your user name cannot be an email address") - if emails: - errors.append("That email has already been used") - if pass_short: - errors.append("Pick a longer password") - if pass_long: - errors.append("Pick a shorter password") - if name_len: - errors.append("Pick a longer user name") - if valid_website is False: - errors.append("Websites must be a proper URL starting with http or https") - if valid_country is False: - errors.append("Invalid country") - if valid_affiliation is False: - errors.append("Please provide a shorter affiliation") - - if len(errors) > 0: -~~ return render_template( - "register.html", - errors=errors, - name=request.form["name"], - email=request.form["email"], - password=request.form["password"], - ) - else: - with app.app_context(): - user = Users(name=name, email=email_address, password=password) - - if website: - user.website = website - if affiliation: - user.affiliation = affiliation - if country: - user.country = country - - db.session.add(user) - db.session.commit() - db.session.flush() - - login_user(user) - - if config.can_send_mail() and get_config( - "verify_emails" - ): # Confirming users is enabled and we can send email. - log( - "registrations", - format="[{date}] {ip} - {name} registered (UNCONFIRMED) with {email}", - ) - email.verify_email_address(user.email) - db.session.close() - return redirect(url_for("auth.confirm")) - else: # Don't care about confirming users - if ( - config.can_send_mail() - ): # We want to notify the user that they have registered. - email.successful_registration_notification(user.email) - - log("registrations", "[{date}] {ip} - {name} registered with {email}") - db.session.close() - - if is_teams_mode(): - return redirect(url_for("teams.private")) - - return redirect(url_for("challenges.listing")) - else: -~~ return render_template("register.html", errors=errors) - - -@auth.route("/login", methods=["POST", "GET"]) -@ratelimit(method="POST", limit=10, interval=5) -def login(): - errors = get_errors() - if request.method == "POST": - name = request.form["name"] - - if validators.validate_email(name) is True: - user = Users.query.filter_by(email=name).first() - else: - user = Users.query.filter_by(name=name).first() + pass + try: + r = client.get("/themes/foo_fallback/static/js/pages/main.dev.js") + except TemplateNotFound: + pass + destroy_ctfd(app) - if user: - if user and verify_password(request.form["password"], user.password): - session.regenerate() + app = create_ctfd() + with app.app_context(): + set_config("ctf_theme", "foo_fallback") + assert app.config["THEME_FALLBACK"] == True + with app.test_client() as client: + r = client.get("/") + assert r.status_code == 200 + r = client.get("/themes/foo_fallback/static/js/pages/main.dev.js") + assert r.status_code == 200 + destroy_ctfd(app) - login_user(user) - log("logins", "[{date}] {ip} - {name} logged in") + os.rmdir(os.path.join(app.root_path, "themes", "foo_fallback")) - db.session.close() - if request.args.get("next") and validators.is_safe_url( - request.args.get("next") - ): - return redirect(request.args.get("next")) - return redirect(url_for("challenges.listing")) - else: - log("logins", "[{date}] {ip} - submitted invalid password for {name}") - errors.append("Your username or password is incorrect") - db.session.close() -~~ return render_template("login.html", errors=errors) - else: - log("logins", "[{date}] {ip} - submitted invalid account information") - errors.append("Your username or password is incorrect") - db.session.close() -~~ return render_template("login.html", errors=errors) - else: - db.session.close() -~~ return render_template("login.html", errors=errors) +def test_theme_template_loading_by_prefix(): + app = create_ctfd() + with app.test_request_context(): + tpl1 = render_template_string("{% extends 'core/page.html' %}", content="test") +~~ tpl2 = render_template("page.html", content="test") + assert tpl1 == tpl2 -@auth.route("/oauth") -def oauth_login(): - endpoint = ( - get_app_config("OAUTH_AUTHORIZATION_ENDPOINT") - or get_config("oauth_authorization_endpoint") - or "https://auth.majorleaguecyber.org/oauth/authorize" - ) - - if get_config("user_mode") == "teams": - scope = "profile team" - else: - scope = "profile" - - client_id = get_app_config("OAUTH_CLIENT_ID") or get_config("oauth_client_id") +def test_theme_template_disallow_loading_admin_templates(): + app = create_ctfd() + with app.app_context(): + try: + filename = os.path.join( + app.root_path, "themes", "foo_disallow", "admin", "malicious.html" + ) + os.makedirs(os.path.dirname(filename), exist_ok=True) + set_config("ctf_theme", "foo_disallow") + with open(filename, "w") as f: + f.write("malicious") + + with pytest.raises(TemplateNotFound): + render_template_string("{% include 'admin/malicious.html' %}") + finally: + shutil.rmtree( + os.path.join(app.root_path, "themes", "foo_disallow"), + ignore_errors=True, + ) - if client_id is None: - error_for( - endpoint="auth.login", - message="OAuth Settings not configured. " - "Ask your CTF administrator to configure MajorLeagueCyber integration.", - ) - return redirect(url_for("auth.login")) ## ... source file continues with no further render_template examples... @@ -571,7 +312,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) backend, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling and many others. flask-base is provided as open source under the @@ -1062,7 +803,7 @@ def update(c_id): is an example application that ties together the [intTellInput.js](https://github.com/jackocnr/intl-tel-input) JavaScript plugin with the -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) form-handling +[Flask-WTF](https://flask-wtf.readthedocs.io/) form-handling library. flask-phone-input is provided as open source under the [MIT license](https://github.com/miguelgrinberg/flask-phone-input/blob/1a1c227c044474ce0fe133493d7f8b0fb8312409/LICENSE). @@ -1115,7 +856,118 @@ def show_phone(): ``` -## Example 10 from flask-restx +## Example 10 from Flask-Meld +[Flask-Meld](https://github.com/mikeabrahamsen/Flask-Meld) +([PyPI package information](https://pypi.org/project/Flask-Meld/)) +allows you to write your front end web code in your back end +Python code. It does this by adding a `{% meld_scripts %}` tag to +the Flask template engine and then inserting components written +in Python scripts created by a developer. + +[**Flask-Meld / flask_meld / component.py**](https://github.com/mikeabrahamsen/Flask-Meld/blob/main/flask_meld/./component.py) + +```python +# component.py +import os +import uuid +from importlib.util import module_from_spec, spec_from_file_location +from itertools import groupby +from operator import itemgetter + +import orjson +from bs4 import BeautifulSoup +from bs4.element import Tag +from bs4.formatter import HTMLFormatter +~~from flask import current_app, jsonify, render_template +from jinja2.exceptions import TemplateNotFound + + +def convert_to_snake_case(s): + s.replace("-", "_") + return s + + +def convert_to_camel_case(s): + s = convert_to_snake_case(s) + return "".join(word.title() for word in s.split("_")) + + +def get_component_class(component_name): + module_name = convert_to_snake_case(component_name) + class_name = convert_to_camel_case(module_name) + module = get_component_module(module_name) + component_class = getattr(module, class_name) + + return component_class + + +def get_component_module(module_name): + user_specified_dir = current_app.config.get("MELD_COMPONENT_DIR", None) + + +## ... source file abbreviated to get to render_template examples ... + + + for func in dir(self) + if callable(getattr(self, func)) + and not func.startswith("_") + and func not in self._meld_attrs + ] + + for func in function_list: + functions[func] = getattr(self, func) + + return functions + + def __context__(self): + return { + "attributes": self._attributes(), + "methods": self._functions(), + } + + def updated(self, name): + pass + + def render(self, component_name: str): + return self._view(component_name) + + def _render_template(self, template_name: str, context_variables: dict): + try: +~~ return render_template(template_name, **context_variables) + except TemplateNotFound: +~~ return render_template(f"meld/{template_name}", **context_variables) + + def _view(self, component_name: str): + data = self._attributes() + context = self.__context__() + context_variables = {} + context_variables.update(context["attributes"]) + context_variables.update(context["methods"]) + context_variables.update({"form": self._form}) + + rendered_template = self._render_template( + f"{component_name}.html", context_variables + ) + + soup = BeautifulSoup(rendered_template, features="html.parser") + root_element = Component._get_root_element(soup) + root_element["meld:id"] = str(self.id) + self._set_values(root_element, context_variables) + + script = soup.new_tag("script", type="module") + init = {"id": str(self.id), "name": component_name, "data": jsonify(data).json} + init_json = orjson.dumps(init).decode("utf-8") + + meld_import = 'import {Meld} from "/meld_js_src/meld.js";' + script.string = f"{meld_import} Meld.componentInit({init_json});" + + +## ... source file continues with no further render_template examples... + +``` + + +## Example 11 from flask-restx [Flask RESTX](https://github.com/python-restx/flask-restx) is an extension that makes it easier to build [RESTful APIs](/application-programming-interfaces.html) into @@ -1170,12 +1022,12 @@ def ui_for(api): ``` -## Example 11 from flaskSaaS +## Example 12 from flaskSaaS [flaskSaas](https://github.com/alectrocute/flaskSaaS) is a boilerplate starter project to build a software-as-a-service (SaaS) web application in [Flask](/flask.html), with [Stripe](/stripe.html) for billing. The boilerplate relies on many common Flask extensions such as -[Flask-WTF](https://flask-wtf.readthedocs.io/en/latest/), +[Flask-WTF](https://flask-wtf.readthedocs.io/), [Flask-Login](https://flask-login.readthedocs.io/en/latest/), [Flask-Admin](https://flask-admin.readthedocs.io/en/latest/), and many others. The project is provided as open source under the @@ -1368,116 +1220,6 @@ def paySuccess(): -## ... source file continues with no further render_template examples... - -``` - - -## Example 12 from Flask-Security-Too -[Flask-Security-Too](https://github.com/Flask-Middleware/flask-security/) -([PyPi page](https://pypi.org/project/Flask-Security-Too/) and -[project documentation](https://flask-security-too.readthedocs.io/en/stable/)) -is a maintained fork of the original -[Flask-Security](https://github.com/mattupstate/flask-security) project that -makes it easier to add common security features to [Flask](/flask.html) -web applications. A few of the critical goals of the Flask-Security-Too -project are ensuring JavaScript client-based single-page applications (SPAs) -can work securely with Flask-based backends and that guidance by the -[OWASP](https://owasp.org/) organization is followed by default. - -The Flask-Security-Too project is provided as open source under the -[MIT license](https://github.com/Flask-Middleware/flask-security/blob/master/LICENSE). - -[**Flask-Security-Too / flask_security / core.py**](https://github.com/Flask-Middleware/flask-security/blob/master/flask_security/./core.py) - -```python -# core.py - -from datetime import datetime, timedelta -import warnings - -import pkg_resources -~~from flask import _request_ctx_stack, current_app, render_template -from flask_babelex import Domain -from flask_login import AnonymousUserMixin, LoginManager -from flask_login import UserMixin as BaseUserMixin -from flask_login import current_user -from flask_principal import Identity, Principal, RoleNeed, UserNeed, identity_loaded -from itsdangerous import URLSafeTimedSerializer -from passlib.context import CryptContext -from werkzeug.datastructures import ImmutableList -from werkzeug.local import LocalProxy - -from .decorators import ( - default_reauthn_handler, - default_unauthn_handler, - default_unauthz_handler, -) -from .forms import ( - ChangePasswordForm, - ConfirmRegisterForm, - ForgotPasswordForm, - LoginForm, - PasswordlessLoginForm, - RegisterForm, - ResetPasswordForm, - SendConfirmationForm, - - -## ... source file abbreviated to get to render_template examples ... - - - sms_service = cv("SMS_SERVICE", app=app) - if sms_service == "Twilio": # pragma: no cover - self._check_modules("twilio", "SMS") - if state.phone_util_cls == PhoneUtil: - self._check_modules("phonenumbers", "SMS") - - secrets = cv("TOTP_SECRETS", app=app) - issuer = cv("TOTP_ISSUER", app=app) - if not secrets or not issuer: - raise ValueError("Both TOTP_SECRETS and TOTP_ISSUER must be set") - state.totp_factory(state.totp_cls(secrets, issuer)) - - if cv("PASSWORD_COMPLEXITY_CHECKER", app=app) == "zxcvbn": - self._check_modules("zxcvbn", "PASSWORD_COMPLEXITY_CHECKER") - return state - - def _check_modules(self, module, config_name): # pragma: no cover - from importlib.util import find_spec - - module_exists = find_spec(module) - if not module_exists: - raise ValueError(f"{module} is required for {config_name}") - - return module_exists - -~~ def render_template(self, *args, **kwargs): -~~ return render_template(*args, **kwargs) - - def render_json(self, cb): - self._state._render_json = cb - - def want_json(self, fn): - self._state._want_json = fn - - def unauthz_handler(self, cb): - self._state._unauthz_handler = cb - - def unauthn_handler(self, cb): - self._state._unauthn_handler = cb - - def reauthn_handler(self, cb): - self._state._reauthn_handler = cb - - def password_validator(self, cb): - self._state._password_validator = cb - - def __getattr__(self, name): - return getattr(self._state, name, None) - - - ## ... source file continues with no further render_template examples... ``` @@ -1807,14 +1549,12 @@ The code is open sourced under the ```python # mathjax.py -from __future__ import absolute_import - ~~from flask import current_app, render_template -class MathjaxMixin(object): +class MathjaxMixin: def _get_head_content(self): -~~ return render_template('mathjax_config.html') + unicode(current_app.manifest['mathjax.js']) +~~ return render_template('mathjax_config.html') + str(current_app.manifest['mathjax.js']) @@ -1858,7 +1598,8 @@ from util import base64_to_pil app = Flask(__name__) -from keras.applications.mobilenet_v2 import MobileNetV2 + +from tensorflow.keras.applications.mobilenet_v2 import MobileNetV2 model = MobileNetV2(weights='imagenet') print('Model loaded. Check http://127.0.0.1:5000/') @@ -1930,6 +1671,7 @@ import configparser import json import logging import os +from typing import Union import requests import requests_cache @@ -1955,15 +1697,16 @@ requests_cache.install_cache(cache_name='news_cache', expire_after=300) APP = Flask(__name__) +SESSION = requests.Session() +SESSION.headers.update({'Authorization': API_KEY}) -@APP.route('/', methods=['GET', 'POST']) -def root(): ## ... source file abbreviated to get to render_template examples ... + return redirect(url_for('category', category='general', page=1)) @APP.route('/category/', methods=['GET', 'POST']) @@ -1978,14 +1721,13 @@ def category(category): country = get_cookie('country') if country is not None: params.update({'country': country}) - response = requests.get(TOP_HEADLINES, - params=params, - headers={'Authorization': API_KEY}) + response = SESSION.get(TOP_HEADLINES, params=params) if response.status_code == 200: pages = count_pages(response.json()) if page > pages: page = pages - return redirect(url_for('category', category=category, page=page)) + return redirect( + url_for('category', category=category, page=page)) articles = parse_articles(response.json()) return render(articles, page, pages, country, category) elif response.status_code == 401: @@ -1994,7 +1736,7 @@ def category(category): @APP.route('/search/', methods=['GET', 'POST']) -def search(query): +def search(query: str): page = request.args.get('page', default=1, type=int) if page < 1: return redirect(url_for('search', query=query, page=1)) @@ -2006,19 +1748,20 @@ def search(query): } if request.method == 'POST': return do_post(page, category='search', current_query=query) - response = requests.get(EVERYTHING, - params=params, - headers={'Authorization': API_KEY}) + response = SESSION.get(EVERYTHING, params=params) pages = count_pages(response.json()) if page > pages: page = pages return redirect(url_for('search', query=query, page=page)) articles = parse_articles(response.json()) + return render(articles, + page, ## ... source file abbreviated to get to render_template examples ... + parsed_articles = [] if response.get('status') == 'ok': for article in response.get('articles'): parsed_articles.append({ @@ -2035,11 +1778,10 @@ def search(query): return parsed_articles -def count_pages(response): - pages = 0 +def count_pages(response: dict) -> int: if response.get('status') == 'ok': - pages = (-(-response.get('totalResults', 0) // PAGE_SIZE)) - return pages + return (-(-response.get('totalResults', 0) // PAGE_SIZE)) + return 0 def render(articles, page, pages, country, category): @@ -2054,9 +1796,8 @@ def render(articles, page, pages, country, category): pages=pages) -def get_cookie(key): - cookie_value = request.cookies.get(key) - return cookie_value +def get_cookie(key: str) -> Union[str, None]: + return request.cookies.get(key) if __name__ == '__main__': @@ -2380,7 +2121,8 @@ import os import requests import yaml -~~from flask import Flask, session, render_template +~~from flask import Flask, render_template +from flask import session as current_session from flask_mail import Mail from flask_migrate import Migrate, MigrateCommand from flask.sessions import SessionInterface @@ -2404,7 +2146,6 @@ def get_config(): app.config.from_object('app.settings') if 'APPLICATION_SETTINGS' in os.environ: app.config.from_envvar(os.environ['APPLICATION_SETTINGS']) - if 'AWS_SECRETS_MANAGER_CONFIG' in os.environ: ## ... source file abbreviated to get to render_template examples ... @@ -2425,7 +2166,7 @@ def get_config(): @user_logged_out.connect_via(app) def clear_session(sender, user, **extra): - session.clear() + current_session.clear() def init_celery_service(app): @@ -2478,6 +2219,7 @@ import os ~~from flask import Flask, render_template, session, request, json, redirect, url_for, send_from_directory from flask_cors import CORS from trape import Trape +import urllib from core.db import Database trape = Trape(1) @@ -2501,7 +2243,6 @@ def index(): return trape.injectCSS_Paths(render_template("/login.html").replace('[LOGIN_SRC]', trape.JSFiles[2]['src']).replace('[LIBS_SRC]', trape.JSFiles[1]['src'])) - ## ... source file abbreviated to get to render_template examples ... diff --git a/content/pages/examples/flask/flask-views-http-method-funcs.markdown b/content/pages/examples/flask/flask-views-http-method-funcs.markdown index b2fec1e84..d9ad8fb08 100644 --- a/content/pages/examples/flask/flask-views-http-method-funcs.markdown +++ b/content/pages/examples/flask/flask-views-http-method-funcs.markdown @@ -19,7 +19,7 @@ and View are a couple of other callables within the `flask.views` package that also have code examples. -You should read up on these subjects along with these `http_method_funcs` examples: +These subjects go along with the `http_method_funcs` code examples: * [web development](/web-development.html) and [web design](/web-design.html) * [web framework concepts](/web-frameworks.html) and the [Flask framework](/flask.html) @@ -45,7 +45,7 @@ from __future__ import unicode_literals import inspect import warnings import logging -from collections import namedtuple +from collections import namedtuple, OrderedDict import six from flask import request diff --git a/content/pages/examples/flask/flask-views-methodview.markdown b/content/pages/examples/flask/flask-views-methodview.markdown index e048933c1..c0e34c8e9 100644 --- a/content/pages/examples/flask/flask-views-methodview.markdown +++ b/content/pages/examples/flask/flask-views-methodview.markdown @@ -19,7 +19,7 @@ and http_method_funcs are a couple of other callables within the `flask.views` package that also have code examples. -These topics are also useful while reading the `MethodView` examples: +You should read up on these subjects along with these `MethodView` examples: * [web development](/web-development.html) and [web design](/web-design.html) * [web framework concepts](/web-frameworks.html) and the [Flask framework](/flask.html) @@ -130,7 +130,7 @@ logger = logging.getLogger(__name__) identifier=form.login.data, secret=form.password.data ) login_user(user, remember=form.remember_me.data) - return redirect_or_next(url_for("forum.index")) + return redirect_or_next(url_for("forum.index"), False) except StopAuthentication as e: flash(e.reason, "danger") except Exception: @@ -430,7 +430,12 @@ from __future__ import unicode_literals from flask import request ~~from flask.views import MethodView -from werkzeug.wrappers import BaseResponse +from werkzeug import __version__ as werkzeug_version + +if werkzeug_version.split('.')[0] >= '2': + from werkzeug.wrappers import Response as BaseResponse +else: + from werkzeug.wrappers import BaseResponse from .model import ModelBase diff --git a/content/pages/examples/sqlalchemy/sqlalchemy-exc-integrityerror.markdown b/content/pages/examples/sqlalchemy/sqlalchemy-exc-integrityerror.markdown index 615febd9c..88658c4f4 100644 --- a/content/pages/examples/sqlalchemy/sqlalchemy-exc-integrityerror.markdown +++ b/content/pages/examples/sqlalchemy/sqlalchemy-exc-integrityerror.markdown @@ -142,7 +142,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) backend, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling and many others. flask-base is provided as open source under the diff --git a/content/pages/examples/sqlalchemy/sqlalchemy-extensions-plug-ins.markdown b/content/pages/examples/sqlalchemy/sqlalchemy-extensions-plug-ins.markdown index 0ff095a54..c24866e5f 100644 --- a/content/pages/examples/sqlalchemy/sqlalchemy-extensions-plug-ins.markdown +++ b/content/pages/examples/sqlalchemy/sqlalchemy-extensions-plug-ins.markdown @@ -49,7 +49,7 @@ libraries that are commonly used in Flask projects, such as [SendGrid](https://www.twilio.com/sendgrid) for transactional email, [SQLAlchemy](/sqlalchemy.html) for persistent data storage through a [relational database](/databases.html) back end, -[Flask-WTF](https://flask-wtf.readthedocs.io/en/stable/) for form +[Flask-WTF](https://flask-wtf.readthedocs.io/) for form handling, and many others. flask-base is provided as open source under the diff --git a/content/pages/meta/00-change-log.markdown b/content/pages/meta/00-change-log.markdown index b2e7a993e..bf6137a13 100644 --- a/content/pages/meta/00-change-log.markdown +++ b/content/pages/meta/00-change-log.markdown @@ -13,10 +13,81 @@ view commit-level changes via the [source repository's commit log](https://github.com/mattmakai/fullstackpython.com/commits/) on GitHub. +## 2022 +### October +* Starting to get back into updating this site again! Note that I'll probably + spend most of my side project time on [Plushcap](https://www.plushcap.com/) + but I'm removing old resources and adding new good ones on here to keep the + site maintained. + +### March +* I decided to go so minimal that I removed the logo to make the site load + faster, and also got rid of some unncessary CSS on front page. + +### February +* Still on break, but updated the header and footer. Also made some website + tweaks to make it faster. + +## 2021 +### October +* Taking a break for a few months to recharge and work on a different coding + side project that I'm excited about. Be back in a few months with some updates. + +### September +* Clean up continues across the site to remove link rot and replace + out-of-date links with newer ones where necessary. + +### August +* New blog post on + [Application Performance Monitoring AWS Lambda Functions with Sentry](/blog/application-performance-monitoring-aws-lambda-functions-sentry.html). + +### June +* Fix broken links across the site. It's a real shame people and companies don't use + 301 redirects more often rather than just breaking their URL structures. + +### May +* New resources on [GPT-3](/gpt-3.html), [Jupyter Notebook](/jupyter-notebook.html) + and [serverless](/serverless.html) pages. + +### April +* Published new post on [How to Monitor Python Functions on AWS Lambda with Sentry](/blog/monitor-python-functions-aws-lambda-sentry.html). +* New [SQLite](/sqlite.html) resource. + +### March +* New [webhook](/webhooks.html) resource. + +### February +* New [GPT-3](/gpt-3.html) resources. + +### January +* Published a new blog post on + [Using Django & AssemblyAI for More Accurate Twilio Call Transcriptions](/blog/django-accurate-twilio-voice-transcriptions.html). +* Updated + [Flask extension projects and example code](/flask-extensions-plug-ins-related-libraries.html). +* Happy New Year! + + ## 2020 +### December +* New [Bash](/bourne-again-shell-bash.html), [tmux](/tmux.html) and + [Vim](/vim.html) resources and explanations. + +### November +* New resources for [DevOps](/devops.html) and [containers](/containers.html). + +### October +* Added new blog on + [Higher Accuracy Twilio Voice Transcriptions with Python and Flask](/blog/accurate-twilio-voice-call-recording-transcriptions-assemblyai.html). +* New [NumPy](/scipy-numpy.html) resources. + +### September +* Tweaks across [Flask code examples](/flask-code-examples.html). + ### August -* New blog post that shows - [How to Transcribe Speech Recordings into Text with Python](/blog/transcribe-recordings-speech-text-assemblyai.html). +* New blog posts on + [How to Transcribe Speech Recordings into Text with Python](/blog/transcribe-recordings-speech-text-assemblyai.html) + and + [Using Sentry to Handle Python Exceptions in Django Projects](/blog/sentry-handle-exceptions-django-projects.html). * Added new [GPT-3 page](/gpt-3.html). * Updated [debugging](/debugging.html) page with some additional descriptions. diff --git a/content/posts/160518-install-postgresql-python-3-ubuntu-1604.markdown b/content/posts/160518-install-postgresql-python-3-ubuntu-1604.markdown index e5d4338cc..baf38465d 100644 --- a/content/posts/160518-install-postgresql-python-3-ubuntu-1604.markdown +++ b/content/posts/160518-install-postgresql-python-3-ubuntu-1604.markdown @@ -12,7 +12,7 @@ headeralt: PostgreSQL and Ubuntu logos. Copyright their respective owners. [PostgreSQL](/postgresql.html) is a powerful open source [relational database](/databases.html) frequently used to create, read, update and delete [Python web application](/web-frameworks.html) data. -[Psycopg2](http://initd.org/psycopg/) is a PostgreSQL database +[Psycopg2](https://www.psycopg.org/) is a PostgreSQL database driver that serves as a Python client for access to the PostgreSQL server. This post explains how to install PostgreSQL on [Ubuntu 16.04](/ubuntu.html) and run a few basic SQL queries within a Python program. @@ -110,7 +110,7 @@ found in the ## Installing psycopg2 Now that PostgreSQL is installed and we have a non-superuser account, we -can install the [psycopg2](http://initd.org/psycopg/) package. Let's +can install the [psycopg2](https://www.psycopg.org/) package. Let's figure out where our `python3` executable is located, create a virtualenv with `python3`, activate the virtualenv and then install the psycopg2 package with `pip`. Find your `python3` executable using the `which` command. diff --git a/content/posts/160626-django-gunicorn-mint-linux-17.markdown b/content/posts/160626-django-gunicorn-mint-linux-17.markdown index 688adaaf8..8ab8836ff 100644 --- a/content/posts/160626-django-gunicorn-mint-linux-17.markdown +++ b/content/posts/160626-django-gunicorn-mint-linux-17.markdown @@ -3,7 +3,7 @@ slug: python-3-django-gunicorn-linux-mint-17 meta: A step-by-step walkthrough on configuring Linux Mint 17.3 with Python 3, Django and Green Unicorn (Gunicorn). category: post date: 2016-06-26 -modified: 2016-07-22 +modified: 2021-06-17 newsletter: False headerimage: /img/160626-mint-django-gunicorn/header.jpg headeralt: Django, Green Unicorn and Linux Mint logos. Copyright their respective owners. @@ -32,7 +32,7 @@ their current versions as of June 2016 are: * [Python](/why-use-python.html) version [3.5.1](https://www.python.org/downloads/release/python-351/) * [Django](/django.html) web framework version - [1.9.7](https://docs.djangoproject.com/en/1.9/releases/1.9/) + [1.9.x](https://pypi.org/project/Django/1.9.13/) * [Green Unicorn (Gunicorn)](/green-unicorn-gunicorn.html) version [19.6](http://docs.gunicorn.org/en/stable/news.html) diff --git a/content/posts/170428-python-2-7-aws-lambda.markdown b/content/posts/170428-python-2-7-aws-lambda.markdown index 77a3e8ec9..c702fc194 100644 --- a/content/posts/170428-python-2-7-aws-lambda.markdown +++ b/content/posts/170428-python-2-7-aws-lambda.markdown @@ -3,7 +3,7 @@ slug: aws-lambda-python-2-7 meta: Learn how to create and deploy your first Amazon Web Services (AWS) Lambda function with Python 2.7. category: post date: 2017-04-28 -modified: 2017-04-29 +modified: 2021-03-30 newsletter: False headerimage: /img/170428-aws-lambda-python-2-7/header.jpg headeralt: AWS, AWS Lambda and Python logos, copyright their respective owners. @@ -22,6 +22,11 @@ function that executes some simple Python 2.7 code and handles environment variables. The code can then be modified to build far more complicated Python applications. +*Note*: AWS +[ended support for Python 2.7 Lambda functions in 2021](https://aws.amazon.com/blogs/compute/announcing-end-of-support-for-python-2-7-in-aws-lambda/) +and Python 2.7 no longer receives support so you should really be using +[Python 3.8 or above](https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-now-supports-python-3-8/). + ## Tools We Need We do not need any local development environment tools to get through diff --git a/content/posts/170723-monitor-flask-apps.markdown b/content/posts/170723-monitor-flask-apps.markdown index b3cd8b760..4ef02bfc6 100644 --- a/content/posts/170723-monitor-flask-apps.markdown +++ b/content/posts/170723-monitor-flask-apps.markdown @@ -165,7 +165,7 @@ The above [Jinja2](/jinja2.html) template is basic HTML without any [embedded template tags](http://jinja.pocoo.org/docs/latest/templates/). The template creates a very plain page with a header description of "PUBG so good" and a GIF from this -[excellent computer game](http://store.steampowered.com/app/578080/PLAYERUNKNOWNS_BATTLEGROUNDS/). +[excellent computer game](https://store.steampowered.com/app/578080/PUBG_BATTLEGROUNDS/). Time to run and test our code. Change into the base directory of your project where `app.py` file is located. Execute `app.py` using the `python` diff --git a/content/posts/190626-dev-led-sales-startups.markdown b/content/posts/190626-dev-led-sales-startups.markdown index 55e4ec431..ba8066e9e 100644 --- a/content/posts/190626-dev-led-sales-startups.markdown +++ b/content/posts/190626-dev-led-sales-startups.markdown @@ -456,9 +456,9 @@ tactics.

Conferences split into a couple of categories: community-run and vendor-run. For example, -WWDC is Apple's vendor-run +WWDC is Apple's vendor-run developer conference. They run the show and control the messaging. -PyCon US is the community-run +PyCon US is the community-run Python developer conference. There are a ton of vendors there as sponsors but no one company controls what happens at the conference.

diff --git a/content/posts/200308-financial-resources-developers.markdown b/content/posts/200308-financial-resources-developers.markdown index fb0619334..80f2560d0 100644 --- a/content/posts/200308-financial-resources-developers.markdown +++ b/content/posts/200308-financial-resources-developers.markdown @@ -77,7 +77,7 @@ macroeconomic data trends. is well-written, straightforward and accessible, particularly because it clearly targets its software developer audience. -* [Don't Quit Your Day Job](https://dqydj.com/) uses a ton of metrics +* [Don't Quit Your Day Job](https://dqydj.com) uses a ton of metrics and statistics to ground their articles on financial topics that are often relevant specifically to software developers. For example, the article on diff --git a/content/posts/200630-report-errors-flask-web-apps-sentry.markdown b/content/posts/200630-report-errors-flask-web-apps-sentry.markdown index e55c14c5c..2e9ec9684 100644 --- a/content/posts/200630-report-errors-flask-web-apps-sentry.markdown +++ b/content/posts/200630-report-errors-flask-web-apps-sentry.markdown @@ -260,5 +260,5 @@ or [@mattmakai](https://twitter.com/mattmakai). I am also on GitHub with the username [mattmakai](https://github.com/mattmakai). If you see an issue or error in this tutorial, please -[fork the source repository on GitHub](https://github.com/mattmakai/fullstackpython/blob/master/content/posts/200630-report-errors-flask-web-apps-sentry.markdown) +[fork the source repository on GitHub](https://github.com/mattmakai/fullstackpython.com/blob/master/content/posts/200630-report-errors-flask-web-apps-sentry.markdown) and submit a pull request with the fix. diff --git a/content/posts/200809-transcribe-recordings-speech-text-assemblyai.markdown b/content/posts/200809-transcribe-recordings-speech-text-assemblyai.markdown index 775de816f..4a5027fbe 100644 --- a/content/posts/200809-transcribe-recordings-speech-text-assemblyai.markdown +++ b/content/posts/200809-transcribe-recordings-speech-text-assemblyai.markdown @@ -3,7 +3,7 @@ slug: transcribe-recordings-speech-text-assemblyai meta: Learn to transcribe speech in recordings like MP3s into text with Python and AssemblyAI's API. category: post date: 2020-08-09 -modified: 2020-08-09 +modified: 2021-09-13 newsletter: False headerimage: /img/headers/python-assemblyai.jpg headeralt: Logos for the implementations used in this blog post. Copyright their respective owners. @@ -38,7 +38,7 @@ comfortable with to work with a database instead of writing SQL... ## Tutorial requirements Throughout this tutorial we are going to use the following dependencies, which we will install in just a moment. Make sure you also have Python 3, -[preferrably 3.6 or newer installed](https://www.python.org/downloads/), +[preferably 3.6 or newer installed](https://www.python.org/downloads/), in your environment: We will use the following dependencies to complete this @@ -137,7 +137,7 @@ Create a new directory named `pytranscribe` to store these files as we write them. Then change into the new project directory. ``` -mkdir pytranscibe +mkdir pytranscribe cd pytranscribe ``` @@ -457,9 +457,7 @@ That's it, we've got our transcription! You may be wondering what to do if the accuracy isn't where you need it to be for your situation. That is where [boosting accuracy for keywords or phrases](https://docs.assemblyai.com/guides/boosting-accuracy-for-keywords-or-phrases) -and -[selecting a model that better matches your data](https://docs.assemblyai.com/guides/transcribing-with-a-different-acoustic-or-custom-language-model) -come in. You can use either of those two methods to boost the accuracy +comes in. You can use either of those two methods to boost the accuracy of your recordings to an acceptable level for your situation. @@ -470,7 +468,7 @@ transcribe recordings with speech into text output. Next, take a look at some of their more advanced documentation that goes beyond the basics in this tutorial: -* [Supported file formats](https://docs.assemblyai.com/overview/supported-file-formats) +* [Supported file formats](https://docs.assemblyai.com/faqs/supported-file-formats) * [Transcribing dual channel/stereo recordings](https://docs.assemblyai.com/guides/transcribing-dual-channel-stereo-recordings) * [Getting speaker labels (speaker diarization)](https://docs.assemblyai.com/guides/getting-speaker-labels-speaker-diarization) diff --git a/content/posts/201009-accurate-twilio-voice-call-recording-transcriptions-assemblyai.markdown b/content/posts/201009-accurate-twilio-voice-call-recording-transcriptions-assemblyai.markdown new file mode 100644 index 000000000..a041498f9 --- /dev/null +++ b/content/posts/201009-accurate-twilio-voice-call-recording-transcriptions-assemblyai.markdown @@ -0,0 +1,569 @@ +title: Higher Accuracy Twilio Voice Transcriptions with Python and Flask +slug: accurate-twilio-voice-call-recording-transcriptions-assemblyai +meta: Use AssemblyAI's speech-to-text service to improve recording transcription accuracy for Twilio Programmable Voice phone calls. +category: post +date: 2020-10-10 +modified: 2020-10-10 +newsletter: False +headerimage: /img/headers/python-assemblyai.jpg +headeralt: Logos for the implementations used in this blog post. Copyright their respective owners. + + +[Twilio's Programmable Voice API](https://www.twilio.com/docs/voice) +is commonly used to initiate and receive phone calls, but the transcription +accuracy for [recordings](https://www.twilio.com/docs/voice/api/recording) +often leaves a lot to be desired. In this tutorial, we'll see how to connect an +outbound phone call powered by the Twilio Voice API with +[AssemblyAI's deep learning transcription API](https://docs.assemblyai.com/overview/getting-started) +to get significantly more accurate speech-to-text output. + + +## Required Tools for this Application +Ensure you have Python 3 installed, because Python 2 reached its +end-of-life at the beginning of 2020 and is no longer supported. +Preferrably, you should have +[Python 3.6 or newer installed](https://www.python.org/downloads/) +in your [development environment](/development-environments.html). +This tutorial will also use: + +We will use the following dependencies to complete this +tutorial: + +* [requests](https://requests.readthedocs.io/), version + [2.24.0](https://pypi.org/project/requests/), for accessing the + [AssemblyAI transcription API](https://docs.assemblyai.com/overview/getting-started) +* [Flask](https://flask.palletsprojects.com/en/1.1.x/), version + [1.1.2](https://pypi.org/project/Flask/1.1.2/), to respond to Twilio's + webhooks +* A [Twilio account](https://www.twilio.com/referral/w9pugq), of which a + free trial version is good enough to test this tutorial +* [Twilio Python helper library](https://pypi.org/project/twilio/), + version [6.45.4](https://pypi.org/project/twilio/6.45.4/) or newer, + for interacting with the [REST API](https://www.twilio.com/docs/usage/api) +* An [AssemblyAI](https://www.assemblyai.com/) account, which you can sign + up for a [free key API access key here](https://app.assemblyai.com/login/) +* [Ngrok](https://ngrok.com/) if you need a localhost tunnel to expose + a public URL that webhooks can send a POST request to + +All code in this blog post is available open source under the MIT license +on GitHub under the +[accurate-twilio-voice-call-recording-transcriptions-assemblyai directory of the blog-code-examples repository](https://github.com/fullstackpython/blog-code-examples). +Use the source code as you desire for your own projects. + + +## Configuring our development environment +Change into the directory where you keep your Python +[virtual environments](/virtual-environments-virtualenvs-venvs.html). +Create a new virtualenv for this project using the following +command. + +Start this Python project by creating a new +[virtual environment](/virtual-environments-virtualenvs-venvs.html) +using the following command. I recommend using a separate directory +such as `~/venvs/` (the tilde is a shortcut for your user's `home` +directory) so that you always know where all your virtualenvs are +located. + +```bash +python3 -m venv ~/venvs/record-transcribe +``` + +Activate the virtualenv with the `activate` shell script: + +```bash +source ~/venvs/record-transcribe/bin/activate +``` + +After the above command is executed, the command prompt will +change so that the name of the virtualenv is prepended to the +original command prompt format, so if your prompt is simply +`$`, it will now look like the following: + +```bash +(record-transcribe) $ +``` + +Remember, you have to activate your virtualenv in every new terminal +window where you want to use dependencies in the virtualenv. + +We can now install the required packages +package into the activated but otherwise empty virtualenv. + +``` +pip install Flask==1.1.2 requests==2.24.0 twilio==6.45.4 +``` + +Look for output similar to the following to confirm the appropriate +packages were installed correctly from PyPI. + +``` +(recordtranscribe) $ pip install Flask==1.1.2 requests==2.24.0 twilio=6.45.4 +Collecting Flask + Using cached https://files.pythonhosted.org/packages/f2/28/2a03252dfb9ebf377f40fba6a7841b47083260bf8bd8e737b0c6952df83f/Flask-1.1.2-py2.py3-none-any.whl +Collecting requests + Using cached https://files.pythonhosted.org/packages/45/1e/0c169c6a5381e241ba7404532c16a21d86ab872c9bed8bdcd4c423954103/requests-2.24.0-py2.py3-none-any.whl +Collecting twilio + Using cached https://files.pythonhosted.org/packages/d0/4e/7c377eb1a1d57f011dc1bee2fee77cf1e9a08407b8d44ea25a187a30c78d/twilio-6.45.4.tar.gz +Collecting Werkzeug>=0.15 (from Flask) + Using cached https://files.pythonhosted.org/packages/cc/94/5f7079a0e00bd6863ef8f1da638721e9da21e5bacee597595b318f71d62e/Werkzeug-1.0.1-py2.py3-none-any.whl +Collecting itsdangerous>=0.24 (from Flask) + Using cached https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl +Collecting click>=5.1 (from Flask) + Using cached https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl +Collecting Jinja2>=2.10.1 (from Flask) + Using cached https://files.pythonhosted.org/packages/30/9e/f663a2aa66a09d838042ae1a2c5659828bb9b41ea3a6efa20a20fd92b121/Jinja2-2.11.2-py2.py3-none-any.whl +Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests) + Using cached https://files.pythonhosted.org/packages/9f/f0/a391d1463ebb1b233795cabfc0ef38d3db4442339de68f847026199e69d7/urllib3-1.25.10-py2.py3-none-any.whl +Collecting idna<3,>=2.5 (from requests) + Using cached https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl +Collecting certifi>=2017.4.17 (from requests) + Using cached https://files.pythonhosted.org/packages/5e/c4/6c4fe722df5343c33226f0b4e0bb042e4dc13483228b4718baf286f86d87/certifi-2020.6.20-py2.py3-none-any.whl +Collecting chardet<4,>=3.0.2 (from requests) + Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl +Collecting six (from twilio) + Using cached https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl +Collecting pytz (from twilio) + Using cached https://files.pythonhosted.org/packages/4f/a4/879454d49688e2fad93e59d7d4efda580b783c745fd2ec2a3adf87b0808d/pytz-2020.1-py2.py3-none-any.whl +Collecting PyJWT>=1.4.2 (from twilio) + Using cached https://files.pythonhosted.org/packages/87/8b/6a9f14b5f781697e51259d81657e6048fd31a113229cf346880bb7545565/PyJWT-1.7.1-py2.py3-none-any.whl +Collecting MarkupSafe>=0.23 (from Jinja2>=2.10.1->Flask) + Using cached https://files.pythonhosted.org/packages/0c/12/37f68957526d1ec0883b521934b4e1b8ff3dd8e4fab858a5bf3e487bcee9/MarkupSafe-1.1.1-cp38-cp38-macosx_10_9_x86_64.whl +Installing collected packages: Werkzeug, itsdangerous, click, MarkupSafe, Jinja2, Flask, urllib3, idna, certifi, chardet, requests, six, pytz, PyJWT, twilio + Running setup.py install for twilio ... done +Successfully installed Flask-1.1.2 Jinja2-2.11.2 MarkupSafe-1.1.1 PyJWT-1.7.1 Werkzeug-1.0.1 certifi-2020.6.20 chardet-3.0.4 click-7.1.2 idna-2.10 itsdangerous-1.1.0 pytz-2020.1 requests-2.24.0 six-1.15.0 twilio-6.45.4 urllib3-1.25.10 + +``` + +We can get started coding the application now that we have all of our +required dependencies installed. + + +## Building our application +Time to dig into the code! We're going to write three source files in +this application: + +* `app.py`: a Flask app that will handle the phone call and recording +* `transcribe.py`: a short Python script to invoke AssemblyAI with the + recording and start the transcription process +* `print_transcription.py`: a script to print the output of the + transcription to the terminal + +Remember that you can get access to all three of the completed files in the +`accurate-twilio-voice-call-recording-transcriptions-assemblyai` directory +of the +[blog-code-examples](https://github.com/fullstackpython/blog-code-examples) +Git repository if you do not want to type or copy from the blog post +itself. + +Create a new directory named `record-transcribe` to store your source files +and change into the new directory. + +``` +mkdir record-transcribe +cd record-transcribe +``` + +Create a new file named `app.py` with the following code: + + +```python +import os +from flask import Flask, request +from twilio.twiml.voice_response import VoiceResponse +from twilio.rest import Client + + +app = Flask(__name__) + +# pulls credentials from environment variables +client = Client() + +BASE_URL = os.getenv("BASE_URL") +twiml_instructions_url = "{}/record".format(BASE_URL) +recording_callback_url = "{}/callback".format(BASE_URL) +twilio_phone_number = os.getenv("TWILIO_PHONE_NUMBER") + + +@app.route("/record", methods=["GET", "POST"]) +def record(): + """Returns TwiML which prompts the caller to record a message""" + # Start our TwiML response + response = VoiceResponse() + + # Use to give the caller some instructions + response.say('Ahoy! Call recording starts now.') + + # Use to record the caller's message + response.record() + + # End the call with + response.hangup() + + return str(response) + + +``` + +There are a couple more functions we'll need to add to `app.py` but first +let's take a look at what the above code does. + +We imported parts of both the Flask and Twilio helper libraries, which will +enable us to programmatically create and control phone calls that Twilio +records. Note that when we instantiate the Twilio helper library with the +empty `Client()` constructor, it automatically looks to read two environment +variables, `TWILIO_ACCOUNT_SID` and `TWILIO_AUTH_TOKEN` to gain appropriate +permissions to your Twilio account. If those two environment variables +are not set with those exact names then you will need to explicitly pass +the Account SID and Auth Token for your account into the constructor. + +After the import are the Flask and Twilio library instantiations. +Then we configure the `BASE_URL` by reading from an environment variable. +In this tutorial the `BASE_URL` will be from Ngrok, but it can also +be your domain where your application is deployed, such as +"https://www.twilio.com". We have not yet set these environment variables, +but we will shortly after we finish writing `app.py`. + +After setting `BASE_URL`, and the three other variables set by environment +variables, we have the `record` function. This function is a +[Flask route](https://hackersandslackers.com/flask-routes/) that +generates the [TwiML](https://www.twilio.com/docs/voice/twiml) +that tells Twilio how to handle a phone call. First, an automated voice +alerts the person who picks up that the phone call is being recorded. Then +the recording starts. Whatever the person on the call says will be recorded +and stored by Twilio. + +Finish `app.py` by adding these two following functions after the +`record` function: + +```python +@app.route("/dial/") +def dial(phone_number): + """Dials an outbound phone call to the number in the URL. Just + as a heads up you will never want to leave a URL like this exposed + without authentication and further phone number format verification. + phone_number should be just the digits with the country code first, + for example 14155559812.""" + call = client.calls.create( + to='+{}'.format(phone_number), + from_=twilio_phone_number, + url=twiml_instructions_url, + ) + print(call.sid) + return "dialing +{}. call SID is: {}".format(phone_number, call.sid) + + +@app.route("/get-recording-url/") +def get_recording_url(call_sid): + recording_urls = "" + call = client.calls.get(call_sid) + for r in call.recordings.list(): + recording_urls="\n".join([recording_urls, r.uri]) + return str(recording_urls) +``` + +The `dial` function creates a Flask route that takes a phone number +input as part of the second level path. Note that in a production +application you *must* have better phone number validation or you +will have a security issue with unsanitized inputs. We are doing +this here to easily grab a phone number as input rather than having +to build a whole user interface with an HTML form just to grab a +phone number. `dial` calls the +[Twilio Voice API](https://www.twilio.com/docs/voice) using our +Twilio account credentials so that we can dial an outbound phone +call to the number sent in through the URL. The `twiml_instructions_url` +should be set to the `record` function URL so that it can give the +proper dialing and recording TwiML instructions for how Twilio's +service should handle dialing the phone call. + +Once we dial the outbound phone call, the +[call SID](https://support.twilio.com/hc/en-us/articles/223180488-What-is-a-Call-SID-) +is printed to the terminal. We'll need that call SID to get the +recording after the call is finished. + +Our `app.py` file is all done. We just need to export our environment +variables for our Twilio credentials. + +[Sign up for Twilio](https://www.twilio.com/referral/w9pugq) or +[log into your existing account](https://www.twilio.com/console). +Once you get to the [Twilio Console](https://www.twilio.com/console), +you can obtain your `TWILIO_ACCOUNT_SID` and `TWILIO_AUTH_TOKEN` on the +right side of the page: + +Twilio Console. + +When you sign up you should have a phone number assigned to your account. +You can use that or +[purchase a new phone number](https://www.twilio.com/console/phone-numbers/search) +to use. + +Set three environment variables with the names `TWILIO_ACCOUNT_SID`, +`TWILIO_AUTH_TOKEN`, and `TWILIO_PHONE_NUMBER` using the `export` command +in your terminal. Make sure to replace the values with your own Account SID, +Auth Token and Twilio phone number. + +```bash +export TWILIO_ACCOUNT_SID=xxxxxxxxxxxxx # found in twilio.com/console +export TWILIO_AUTH_TOKEN=yyyyyyyyyyyyyy # found in twilio.com/console +export TWILIO_PHONE_NUMBER=+17166382453 # replace with your Twilio number +``` + +Note that you must use the `export` command in every command line window +that you want this key to be accessible. The scripts we are writing will +not be able to access the Twilio APIs if you do not have the tokens exported +in the environment where you are running the script. + +There is one more environment variable to set before we can run `app.py`. +We need to use Ngrok as a localhost tunnel so that Twilio's webhook can +send an HTTP POST request to our `app.py` Flask application running on +our local development environment. + +Run Ngrok in a new terminal window, because you will need to keep it +running while we run our other Python code: + +```bash +./ngrok http 5000 +``` + +Ngrok running with a localhost tunnel. + +Copy the HTTPS version of the "Forwarding" URL and set the `BASE_URL` +environment variable value to it. For example, in this screenshot you +would set `BASE_URL` to `https://7f9139eaf445.ngrok.io` using the +following command: + +```bash +export BASE_URL=https://7f9139eaf445.ngrok.io # use your ngrok URL, or domain. no trailing slash +``` + +Okay, we can finally run `app.py`. Make sure you are still running Ngrok +in a different window, your virtualenv is active and that in this terminal +you have your four environment variables set, then run the `flask run` +command: + +```bash +flask run +``` + +You should see Flask output something like the following text: + +```bash + * Environment: production + WARNING: This is a development server. Do not use it in a production deployment. + Use a production WSGI server instead. + * Debug mode: off + * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) +``` + +That is a legitimate warning: only use this command for +development purposes and when you want to [deploy](/deployment.html) +to production you need to use a real [WSGI server](/wsgi-servers.html) +like [Gunicorn](/green-unicorn-gunicorn.html). + +Time to test out our application. + + +## Testing Twilio Programmable Voice Recording +We can test our application by going to localhost on port 5000. +Go to this URL in your web browser, replacing the "14155551234" +with the phone number you want to call, where the person on the +line will be recorded: http://localhost:5000/dial/14155551234. + +That number should now receive a phone call from your Twilio +number. Pick up, record a message that you want to use to test +the transcription, and then hang up. + +If you get an error, make sure all of your environment variables +are set. You can check the values by using the echo command like +this: + +```bash +echo $BASE_URL +``` + +When the call is over, copy the call SID show on the web page +so that we can use it to look up where the recording audio +file is stored. + +Twilio call SID. + +Go to "localhost:5000/get-recording-url/" with the call SID +at the end. For example, +"localhost:5000/get-recording-url/CAda3f2f49ff4e8ef2be6b726edb998c92". + +Twilio call recording URL. + +Copy the entire output except for the ".json" at the end, then paste +it into the web browser's URL bar, prepended with "api.twilio.com". +For example, +"https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300". +This will bring up the recording. Copy the entire URL and we will use it +as input into the AssemblyAI service. + + +## Transcribing with the AssemblyAI API +We can now use the AssemblyAI API for speech-to-text transcription on +the call recording that was just made. + +[Sign up for an AssemblyAI account](https://app.assemblyai.com/login/) +and log in to the +[AssemblyAI dashboard](https://app.assemblyai.com/dashboard/), then +copy "Your API token" as shown in this screenshot: + +AssemblyAI dashboard. + +We need to export our AssemblyAI API key as an environment variable +so that our Python application can use it to authenticate with their +API. We also need to pass the publicly-accessible URL for the recording, +so we'll set that as an environment variable as well. + +```bash +# make sure to replace this URL with the one for your recording +export ASSEMBLYAI_KEY=your-api-key-here +export RECORDING_URL=https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300 +``` + +Create a new file named `transcribe.py` and write the following code in it: + +```python +import os +import requests + +endpoint = "https://api.assemblyai.com/v2/transcript" + +json = { + "audio_url": os.getenv("RECORDING_URL") +} + +headers = { + "authorization": os.getenv("ASSEMBLYAI_KEY"), + "content-type": "application/json" +} + +response = requests.post(endpoint, json=json, headers=headers) + +print(response.json()) +``` + +The above code calls the AssemblyAI transcription service using +the secret key and passes it the URL with the file recording. +The script prints out the JSON response from the service, +which will contain a transcription ID that we'll use to access +the results after they finish processing. + +Run the script using the `python` command: + +```bash +python transcribe.py +``` + +You will get back some JSON as output, similar what you see here: + +```bash +{'audio_end_at': None, 'acoustic_model': 'assemblyai_default', 'text': None, 'audio_url': 'https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300', 'speed_boost': False, 'language_model': 'assemblyai_default', 'redact_pii': False, 'confidence': None, 'webhook_status_code': None, 'id': 'zibe9vwmx-82ce-476c-85a7-e82c09c67daf', 'status': 'queued', +'boost_param': None, 'words': None, 'format_text': True, 'webhook_url': None, 'punctuate': True, 'utterances': None, 'audio_duration': None, 'auto_highlights': False, 'word_boost': [], 'dual_channel': None, 'audio_start_from': None} +``` + +Find the value contained with the `id` field of the JSON response. We need +that value to look up the final result of our transcription. Copy the +transcription ID and set it as an environment variable to use as input by +the final script: + +``` +# replace with what's found within `id` from the JSON response +export TRANSCRIPTION_ID=aksd19vwmx-82ce-476c-85a7-e82c09c67daf +``` + +We just need a little more Python that looks up the result and we'll be all +done. + + +## Retrieve the AssemblyAI Transcription +AssemblyAI will be busy transcribing the recording. Depending on the size of +the file it can take anywhere from a few seconds to a few minutes for the +job to complete. We can use the following code to see if the job is still +in progress or it has completed. If the transcription is done it will print +the results to the terminal. + +Create a new file named `print_transcription.py` with the following code: + +```python +import os +import requests + +endpoint = "https://api.assemblyai.com/v2/transcript/{}".format(os.getenv("TRANSCRIPTION_ID")) + +headers = { + "authorization": os.getenv("ASSEMBLYAI_KEY"), +} + +response = requests.get(endpoint, headers=headers) + +print(response.json()) +print("\n\n") +print(response.json()['text']) +``` + +The code above in `print_transcription.py` is very similar to the code +in the previous `transcribe.py` source file. imports `os` (operating system) +from the Python standard library, as we did in the previous two files, +to obtain the `TRANSCRIPTION_ID` and `ASSEMBLYAI_KEY` environment variable +values. + +The `endpoint` is simply the AssemblyAI API endpoint for retrieving +transcriptions. We set the appropriate `authorization` header and +make the API call using the `requests.get` function. We then print +out the JSON response as well as just the text that was transcribed. + +Time to test out this third file. Execute the following command in +the terminal: + +```bash +python print_transcription.py +``` + +Your output will be different based on your recording but you should see a +result in the terminal similar to the following: + +```bash +{'audio_end_at': None, 'acoustic_model': 'assemblyai_default', 'auto_highlights_result': None, 'text': 'An object relational mapper is a code library that automates the transfer of data stored in a relational database tables into objects that are more commonly used in application. Code or MS provide a high level abstraction upon a relational database that allows the developer to write Python code. Instead of sequel to create read update and delete data and schemas in their database developers can use the programming language that they are comfortable with comfortable to work with the database instead of writing sequel statements or short procedures.', 'audio_url': 'https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300', 'speed_boost': False, 'language_model': 'assemblyai_default', 'id': 'zibe9vwmx-82ce-476c-85a7-e82c09c67daf', 'confidence': 0.931797752808989, 'webhook_status_code': None, 'status': 'completed', 'boost_param': None, 'redact_pii': False, 'words': [{'text': 'An', 'confidence': 1.0, 'end': 90, 'start': 0}, {'text': 'object', 'confidence': 0.94, 'end': 570, 'start': 210}, {'text': 'relational', 'confidence': 0.89, 'end': 1080, 'start': 510}, {'text': 'mapper', 'confidence': 0.97, 'end': 1380, 'start': 1020}, {'text': 'is', 'confidence': 0.88, 'end': 1560, 'start': 1350}, {'text': 'a', 'confidence': 0.99, 'end': 1620, 'start': 1500}, {'text': 'code', 'confidence': 0.93, 'end': 1920, 'start': 1620}, {'text': 'library', 'confidence': 0.94, 'end': 2250, 'start': 1860}, {'text': 'that', 'confidence': 0.99, 'end': 2490, 'start': 2220}, {'text': 'automates', 'confidence': 0.93, 'end': 2940, 'start': 2430}, {'text': 'the', 'confidence': 0.95, 'end': 3150, 'start': 2910}, {'text': 'transfer', 'confidence': 0.98, 'end': 3510, 'start': 3090}, {'text': 'of', 'confidence': +0.99, 'end': 3660, 'start': 3480}, {'text': 'data', 'confidence': 0.84, 'end': 3960, 'start': 3630}, {'text': 'stored', 'confidence': 0.89, 'end': 4350, 'start': 3900}, {'text': 'in', 'confidence': 0.98, 'end': 4500, 'start': 4290}, {'text': 'a', 'confidence': 0.85, 'end': 4560, 'start': 4440}, {'text': 'relational', 'confidence': 0.87, 'end': 5580, 'start': 4500}, {'text': 'database', 'confidence': 0.92, 'end': +6030, 'start': 5520}, {'text': 'tables', 'confidence': 0.93, 'end': 6330, 'start': 5970}, {'text': 'into', 'confidence': 0.92, 'end': 7130, 'start': 6560}, {'text': 'objects', 'confidence': 0.96, 'end': 7490, 'start': 7100}, {'text': 'that', 'confidence': 0.97, 'end': 7700, 'start': 7430}, {'text': 'are', 'confidence': 0.9, 'end': 7850, 'start': 7640}, {'text': 'more', 'confidence': 0.97, 'end': 8030, 'start': 7790}, {'text': 'commonly', 'confidence': 0.92, 'end': 8480, 'start': 7970}, {'text': 'used', 'confidence': 0.86, 'end': 8750, 'start': 8420}, {'text': 'in', 'confidence': 0.94, 'end': 9050, 'start': 8840}, {'text': 'application.', 'confidence': 0.98, 'end': 9860, 'start': 9110}, {'text': 'Code', 'confidence': 0.93, 'end': 10040, 'start': 9830}, {'text': 'or', 'confidence': 1.0, 'end': 11210, 'start': 10220}, {'text': 'MS', 'confidence': 0.83, 'end': 11480, 'start': 11180}, {'text': 'provide', 'confidence': 0.94, 'end': 11870, 'start': 11510}, {'text': 'a', 'confidence': 1.0, 'end': 11960, 'start': 11840}, {'text': 'high', 'confidence': 1.0, 'end': 12200, 'start': 11930}, {'text': 'level', 'confidence': 0.94, 'end': 12440, 'start': 12170}, {'text': 'abstraction', 'confidence': 0.95, 'end': 12980, 'start': 12410}, {'text': +'upon', 'confidence': 0.94, 'end': 13220, 'start': 12950}, {'text': 'a', 'confidence': 1.0, 'end': 13280, 'start': 13160}, {'text': 'relational', 'confidence': 0.94, 'end': 13820, 'start': 13280}, {'text': 'database', 'confidence': 0.95, 'end': 14210, 'start': 13790}, {'text': 'that', 'confidence': 0.96, 'end': 14420, 'start': 14150}, {'text': 'allows', 'confidence': 0.99, 'end': 14720, 'start': 14360}, {'text': +'the', 'confidence': 0.56, 'end': 14870, 'start': 14690}, {'text': 'developer', 'confidence': 0.98, 'end': 15290, 'start': 14810}, {'text': 'to', 'confidence': 0.94, 'end': 15410, 'start': 15230}, {'text': 'write', 'confidence': 0.96, 'end': 15680, 'start': 15380}, {'text': 'Python', 'confidence': 0.94, 'end': 16070, 'start': 15620}, {'text': 'code.', 'confidence': 0.98, 'end': 16310, 'start': 16070}, {'text': 'Instead', 'confidence': 0.97, 'end': 17160, 'start': 16500}, {'text': 'of', 'confidence': 0.93, 'end': 17340, 'start': 17130}, {'text': 'sequel', 'confidence': 0.86, 'end': 17820, 'start': 17280}, {'text': 'to', 'confidence': 0.91, 'end': 18090, 'start': 17880}, {'text': 'create', 'confidence': 0.89, 'end': 18450, 'start': 18090}, {'text': 'read', 'confidence': 0.88, 'end': 18840, 'start': 18480}, {'text': 'update', 'confidence': 0.92, 'end': 19290, 'start': 18870}, {'text': 'and', 'confidence': 0.94, 'end': 19590, 'start': 19230}, {'text': 'delete', 'confidence': 0.89, 'end': 19920, 'start': 19530}, {'text': 'data', +'confidence': 0.95, 'end': 20190, 'start': 19890}, {'text': 'and', 'confidence': 0.92, 'end': 20490, 'start': 20250}, {'text': 'schemas', 'confidence': 0.86, 'end': 21000, 'start': 20430}, {'text': 'in', 'confidence': 0.94, 'end': 21210, 'start': 21000}, {'text': 'their', 'confidence': 0.98, 'end': 21510, 'start': 21150}, {'text': 'database', 'confidence': 0.97, 'end': 21900, 'start': 21450}, {'text': 'developers', 'confidence': 0.83, 'end': 23200, 'start': 22420}, {'text': 'can', 'confidence': 0.95, 'end': 23440, 'start': 23200}, {'text': 'use', 'confidence': 0.97, 'end': 23650, 'start': 23410}, {'text': 'the', 'confidence': 0.99, 'end': 23890, 'start': 23590}, {'text': 'programming', 'confidence': 0.97, 'end': 24370, 'start': 23830}, {'text': 'language', 'confidence': 1.0, 'end': 24700, 'start': 24310}, {'text': 'that', 'confidence': 1.0, 'end': 24880, 'start': 24640}, {'text': 'they', 'confidence': 0.99, 'end': 25060, 'start': 24820}, {'text': 'are', 'confidence': 0.85, 'end': 25210, 'start': 25000}, {'text': 'comfortable', 'confidence': 0.92, 'end': 25780, 'start': 25180}, {'text': 'with', 'confidence': 1.0, 'end': 25960, 'start': 25720}, {'text': 'comfortable', 'confidence': 0.94, 'end': 29090, 'start': 28090}, {'text': 'to', 'confidence': 0.84, 'end': 29840, 'start': 29180}, {'text': 'work', 'confidence': 0.95, 'end': 30050, 'start': 29780}, {'text': 'with', 'confidence': 0.98, 'end': 30290, 'start': 30020}, {'text': 'the', 'confidence': 0.69, 'end': 30440, 'start': 30230}, {'text': 'database', 'confidence': 0.98, 'end': 30860, 'start': 30380}, {'text': 'instead', 'confidence': 1.0, 'end': 32780, 'start': 31780}, {'text': 'of', 'confidence': 0.98, 'end': 32900, 'start': 32720}, {'text': 'writing', 'confidence': 0.87, 'end': 33320, 'start': 32870}, {'text': 'sequel', 'confidence': 0.88, 'end': 33860, 'start': 33290}, {'text': 'statements', 'confidence': 0.95, 'end': 34310, 'start': 33800}, {'text': 'or', 'confidence': 0.9, 'end': 34460, 'start': 34250}, {'text': 'short', 'confidence': 0.9, 'end': 34790, 'start': 34430}, {'text': 'procedures.', 'confidence': 0.98, 'end': 35270, 'start': 34760}], 'format_text': True, 'webhook_url': None, 'punctuate': True, 'utterances': None, 'audio_duration': 36.288, 'auto_highlights': False, 'word_boost': [], +'dual_channel': None, 'audio_start_from': None} + + +An object relational mapper is a code library that automates the transfer of data stored in a relational database tables into objects that are more commonly used in application. Code or MS provide a high level abstraction upon a relational database that allows the developer to write Python code. Instead of sequel to create read update and delete data and schemas in their database developers can use the programming language that they are comfortable with comfortable to work with the database instead of writing sequel statements or short procedures. +``` + +That's a lot of output. The first part contains the results of the +transcription and the confidence in the accuracy of each word transcribed. +The second part is just the plain text output from the transcription. + +You can take this now take this base code and add it to any application +that needs high quality text-to-speech transcription. If the results +aren't quite good enough for you, check out this tutorial on +[boosting accuracy for keywords or phrases](https://docs.assemblyai.com/guides/boosting-accuracy-for-keywords-or-phrases) +as well as +[better matching your data with topic detection](https://docs.assemblyai.com/guides/iab-categorization). + + +## What now? +We just finished building a highly accurate transcription application for recordings. + +Next, try out some of these other related Python tutorials: + +* [How to Transcribe Speech Recordings into Text with Python](/blog/transcribe-recordings-speech-text-assemblyai.html) +* [Reporting Exceptions in Python Scripts with Sentry](/blog/report-exceptions-python-scripts-sentry.html) +* [Basic Data Types in Python: Strings](/blog/python-basic-data-types-strings.html) + +Questions? Let me know via +[a GitHub issue ticket on the Full Stack Python repository](https://github.com/mattmakai/fullstackpython.com/issues), +on Twitter +[@fullstackpython](https://twitter.com/fullstackpython) +or [@mattmakai](https://twitter.com/mattmakai). +If you see an issue or error in this tutorial, please +[fork the source repository on GitHub](https://github.com/mattmakai/fullstackpython.com/blob/master/content/posts/201009-accurate-twilio-voice-call-recording-transcriptions-assemblyai.markdown) +and submit a pull request with the fix. + diff --git a/content/posts/210105-django-accurate-twilio-voice-transcriptions.markdown b/content/posts/210105-django-accurate-twilio-voice-transcriptions.markdown new file mode 100644 index 000000000..c4b7d0018 --- /dev/null +++ b/content/posts/210105-django-accurate-twilio-voice-transcriptions.markdown @@ -0,0 +1,716 @@ +title: Using Django & AssemblyAI for More Accurate Twilio Call Transcriptions +slug: django-accurate-twilio-voice-transcriptions +meta: Use Python, Django and AssemblyAI's transcription API to improve recording accuracy for Twilio Programmable Voice phone calls. +category: post +date: 2021-01-05 +modified: 2021-09-13 +newsletter: False +headerimage: /img/headers/django-assemblyai.jpg +headeralt: Logos for the implementations used in this blog post. Copyright their respective owners. + + +[Recording phone calls](https://www.twilio.com/docs/voice/tutorials/how-to-record-phone-calls-python) +with one or more participants is easy with +[Twilio's Programmable Voice API](https://www.twilio.com/docs/voice/quickstart/python), +but the speech-to-text accuracy can be poor, especially for transcription +of words from niche domains such as healthcare and engineering. +[AssemblyAI's API for transcription](https://www.assemblyai.com/) +provides much higher accuracy by default and through optional keyword lists. +accuracy for [recordings](https://www.twilio.com/docs/voice/api/recording). + +In this tutorial, we'll record an outbound Twilio call recording to AssemblyAI's +API to get significantly more accurate speech-to-text output. + + +## Tutorial Prerequisites +Ensure you have Python 3 installed, because Python 2 reached its +end-of-life at the beginning of 2020 and is no longer supported. +Preferrably, you should have +[Python 3.7 or greater installed](https://www.python.org/downloads/) +in your [development environment](/development-environments.html). +This tutorial will also use: + +We will use the following dependencies to complete this +tutorial: + +* [Django](/django.html) version 3.1.x, where *x* is the latest security + release +* A [Twilio account](https://www.twilio.com/referral/w9pugq) and the + [Python Twilio helper library](https://pypi.org/project/twilio/) + version 6.45.2 or newer +* [requests](https://requests.readthedocs.io/) + [version 2.24.0](https://pypi.org/project/requests/) +* An [AssemblyAI](https://www.assemblyai.com/) account, which you can sign up for a [free key API access key here](https://app.assemblyai.com/login/) + +All code in this blog post is available open source under the MIT license +on GitHub under the +[django-accurate-twilio-voice-transcriptions directory of the blog-code-examples repository](https://github.com/fullstackpython/blog-code-examples). +Use the source code as you desire for your own projects. + + +## Configuring our development environment +Change into the directory where you keep your Python +[virtual environments](/virtual-environments-virtualenvs-venvs.html). +Create a new virtualenv for this project using the following +command. + +Start the Django project by creating a new +[virtual environment](/virtual-environments-virtualenvs-venvs.html) +using the following command. I recommend using a separate directory +such as `~/venvs/` (the tilde is a shortcut for your user's `home` +directory) so that you always know where all your virtualenvs are +located. + +```bash +python3 -m venv ~/venvs/djtranscribe +``` + +Activate the virtualenv with the `activate` shell script: + +```bash +source ~/venvs/djtranscribe/bin/activate +``` + +After the above command is executed, the command prompt will +change so that the name of the virtualenv is prepended to the +original command prompt format, so if your prompt is just +`$`, it will now look like the following: + +```bash +(djtranscribe) $ +``` + +Remember, you have to activate your virtualenv in every new terminal +window where you want to use dependencies in the virtualenv. + +We can now install the [Django](https://pypi.org/project/Django/) +package into the activated but otherwise empty virtualenv. + +``` +pip install django==3.1.3 requests==2.24.0 twilio==6.45.2 +``` + +Look for output similar to the following to confirm the appropriate +packages were installed correctly from PyPI. + +``` +(djtranscribe) $ pip install django==3.1.3 requests==2.24.0 twilio=6.45.2 +pip install django requests twilio +Collecting django + Downloading Django-3.1.3-py3-none-any.whl (7.8 MB) + |████████████████████████████████| 7.8 MB 2.6 MB/s +Collecting requests + Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB) +Collecting twilio + Downloading twilio-6.47.0.tar.gz (460 kB) + |████████████████████████████████| 460 kB 19.6 MB/s +Collecting sqlparse>=0.2.2 + Downloading sqlparse-0.4.1-py3-none-any.whl (42 kB) + |████████████████████████████████| 42 kB 4.8 MB/s +Collecting pytz + Downloading pytz-2020.4-py2.py3-none-any.whl (509 kB) + |████████████████████████████████| 509 kB 31.0 MB/s +Collecting asgiref<4,>=3.2.10 + Downloading asgiref-3.3.0-py3-none-any.whl (19 kB) +Collecting chardet<4,>=3.0.2 + Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB) +Collecting idna<3,>=2.5 + Using cached idna-2.10-py2.py3-none-any.whl (58 kB) +Collecting certifi>=2017.4.17 + Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB) +Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 + Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB) + |████████████████████████████████| 127 kB 24.5 MB/s +Collecting six + Using cached six-1.15.0-py2.py3-none-any.whl (10 kB) +Collecting PyJWT>=1.4.2 + Using cached PyJWT-1.7.1-py2.py3-none-any.whl (18 kB) +Using legacy 'setup.py install' for twilio, since package 'wheel' is not installed. +Installing collected packages: sqlparse, pytz, asgiref, django, chardet, idna, certifi, urllib3, requests, six, PyJWT, twilio + Running setup.py install for twilio ... done +Successfully installed PyJWT-1.7.1 asgiref-3.3.0 certifi-2020.6.20 chardet-3.0.4 django-3.1.3 idna-2.10 pytz-2020.4 requests-2.24.0 six-1.15.0 sqlparse-0.4.1 twilio-6.47.0 urllib3-1.25.11 + +``` + +We can get started coding the application now that we have all of our +required dependencies installed. + + +## Starting our Django project +Let's begin coding our application. + +We can use the [Django](/django.html) `django-admin` tool to create +the boilerplate code structure to get our project started. +Change into the directory where you develop your applications. For +example, I typically use `/Users/matt/devel/py/` for all of my +Python projects. Then run the following command to start a Django +project named `djtranscribe`: + +``` +django-admin.py startproject djtranscribe +``` + +Note that in this tutorial we are using the same name for both the +virtualenv and the Django project directory, but they can be +different names if you prefer that for organizing your own projects. + +The `django-admin` command creates a directory named `djtranscribe` +along with several subdirectories that you should be familiar with +if you have previously worked with Django. + +Change directories into the new project. + +``` +cd djtranscribe +``` + +Create a new Django app within `djtranscribe` named `caller`. + +``` +python manage.py startapp caller +``` + +Django will generate a new folder named `caller` in the project. +We should update the URLs so the app is accessible before we write +our `views.py` code. + +Open `djtranscribe/djtranscribe/urls.py`. Add the highlighted +lines so that URL resolver will check the `caller` app +for additional routes to match with URLs that are requested of +this Django application. + +```python +# djtranscribe/djtranscribe/urls.py +~~from django.conf.urls import include +from django.contrib import admin +from django.urls import path + + +urlpatterns = [ +~~ path('', include('caller.urls')), + path('admin/', admin.site.urls), +] +``` + +Save `djtranscribe/djtranscribe/urls.py` and open +`djtranscribe/djtranscribe/settings.py`. +Add the `caller` app to `settings.py` by inserting +the highlighted line: + +```python +# djtranscribe/djtranscribe/settings.py +# Application definition + +INSTALLED_APPS = [ + 'django.contrib.admin', + 'django.contrib.auth', + 'django.contrib.contenttypes', + 'django.contrib.sessions', + 'django.contrib.messages', + 'django.contrib.staticfiles', +~~ 'caller', +] +``` + +Make sure you change the default `DEBUG` and `SECRET_KEY` +values in `settings.py` before you deploy any code to production. Secure +your app properly with the information from the Django +[production deployment checklist](https://docs.djangoproject.com/en/stable/howto/deployment/checklist/) +so that you do not add your project to the list of hacked applications +on the web. + +Save and close `settings.py`. + +Next change into the `djtranscribe/caller` directory. Create +a new file named `urls.py` to contain routes for the `caller` app. + +Add all of these lines to the empty `djtranscribe/caller/urls.py` +file. + +```python +# djtranscribe/caller/urls.py +from django.conf.urls import url +from . import views + +urlpatterns = [ + url(r'', views.index, name="index"), +] +``` + +Save `djtranscribe/caller/urls.py`. Open +`djtranscribe/caller/views.py` to add the +following two highlighted lines. + +``` +# djtranscribe/caller/views.py +from django.http import HttpResponse + + +~~def index(request): +~~ return HttpResponse('Hello, world!', 200) +``` + +We can test out that this simple boilerplate response is +correct before we start adding the meat of the functionality to +the project. Change into the base directory of your Django project +where the `manage.py` file is located. Execute the development +server with the following command: + +```bash +python manage.py runserver +``` + +The Django development server should start up with no issues other than +an unapplied migrations warning. + +``` +Watching for file changes with StatReloader +Performing system checks... + +System check identified no issues (0 silenced). + +November 15, 2020 - 14:07:03 +Django version 3.1.3, using settings 'djtranscribe.settings' +Starting development server at http://127.0.0.1:8000/ +Quit the server with CONTROL-C. +``` + +Open a web browser and go to `localhost:8000`. + +Web browser rendering simple text 'Hello, world!'. + +You should see 'Hello, world!' rendered in the browser. +That means everything is working properly so far and we can +now add the dialing, recording and transcribing capabilities to +our Django project. + + +## Adding Twilio to the Django project +Time to add Twilio's Voice API into the mix so we can dial +a phone call from our Django project and make a recording +out of it. + + +Start by opening up `djtranscribe/djtranscribe/settings.py` +and modifying it with the following highlighted `import os` +line: + +```python +# djtranscribe/djtranscribe/settings.py +~~import os +from pathlib import Path + + +# Build paths inside the project like this: BASE_DIR / 'subdir'. +BASE_DIR = Path(__file__).resolve().parent.parent +``` + +Then at the bottom of the `settings.py` file, add the +following highlighted lines, which will be settings that are pulled from +environment variables we will configure later. + +```python +# Static files (CSS, JavaScript, Images) +# https://docs.djangoproject.com/en/3.1/howto/static-files/ + +STATIC_URL = '/static/' + + +~~BASE_URL = os.getenv("BASE_URL") +~~TWIML_INSTRUCTIONS_URL = "{}/record/".format(BASE_URL) +~~TWILIO_PHONE_NUMBER = os.getenv("TWILIO_PHONE_NUMBER") +``` + +Save `settings.py` and change into the `caller` Django app directory. + +Update `djtranscribe/caller/urls.py` with the the following new +code: + +```python +# djtranscribe/caller/urls.py +from django.conf.urls import url +from . import views + +urlpatterns = [ +~~ url(r'dial/(\d+)/$', views.dial, name="dial"), +~~ url(r'record/$', views.record_twiml, name="record-twiml"), +~~ url(r'get-recording-url/([A-Za-z0-9]+)/$', views.get_recording_url, +~~ name='recording-url'), +] +``` + +Next, open `djtranscribe/views.py` and update it with the following +code, replacing what already exists within the file: + +```python +# djtranscribe/caller/views.py +from django.conf import settings +from django.http import HttpResponse +from django.views.decorators.csrf import csrf_exempt + +from twilio.rest import Client +from twilio.twiml.voice_response import VoiceResponse + + +def dial(request, phone_number): + """Dials an outbound phone call to the number in the URL. Just + as a heads up you will never want to leave a URL like this exposed + without authentication and further phone number format verification. + phone_number should be just the digits with the country code first, + for example 14155559812.""" + # pulls credentials from environment variables + twilio_client = Client() + call = twilio_client.calls.create( + to='+{}'.format(phone_number), + from_=settings.TWILIO_PHONE_NUMBER, + url=settings.TWIML_INSTRUCTIONS_URL, + ) + print(call.sid) + return HttpResponse("dialing +{}. call SID is: {}".format( + phone_number, call.sid)) + + +@csrf_exempt +def record_twiml(request): + """Returns TwiML which prompts the caller to record a message""" + # Start our TwiML response + response = VoiceResponse() + + # Use to give the caller some instructions + response.say('Ahoy! Call recording starts now.') + + # Use to record the caller's message + response.record() + + # End the call with + response.hangup() + + return HttpResponse(str(response), content_type='application/xml') + + +def get_recording_url(request, call_sid): + """Returns an HttpResponse with plain text of the link to one or more + recordings from the specified Call SID.""" + # pulls credentials from environment variables + twilio_client = Client() + recording_urls = "" + call = twilio_client.calls.get(call_sid) + for r in call.recordings.list(): + recording_urls="\n".join([recording_urls, "".join(['https://api.twilio.com', r.uri])]) + return HttpResponse(str(recording_urls), 200) +``` + +Each of the above view functions performs one of the steps needed to +create a call recording of a phone call dialed by Twilio, and then +retrieve it as a file. `dial` programmatically initiates the outbound +call, `record_twiml` returns instructions to play a message that the +call is being recorded, records it, and then hangs up when the call +is done. `get_recording_url` only returns the URL location of the +recorded phone call so that in the next step we can send the file over +to AssemblyAI. + +Our Django project modifications are done. Next, we need to use +two services, Twilio and Ngrok, to enable some of the machine +to happen of phone calling and running the application from our +local machine. + + +## Twilio credentials and environment variables +[Sign up for Twilio](https://www.twilio.com/referral/w9pugq) or +[log into your existing account](https://www.twilio.com/console). +Once you get to the [Twilio Console](https://www.twilio.com/console), +you can obtain your `TWILIO_ACCOUNT_SID` and `TWILIO_AUTH_TOKEN` on the +right side of the page: + +Twilio Console. + +When you sign up you should have a phone number assigned to your account. +You can use that or +[purchase a new phone number](https://www.twilio.com/console/phone-numbers/search) +to use. + +Set three environment variables with the names `TWILIO_ACCOUNT_SID`, +`TWILIO_AUTH_TOKEN`, and `TWILIO_PHONE_NUMBER` using the `export` command +in your terminal. Make sure to replace the values with your own Account SID, +Auth Token and Twilio phone number. + +```bash +export TWILIO_ACCOUNT_SID=xxxxxxxxxxxxx # found in twilio.com/console +export TWILIO_AUTH_TOKEN=yyyyyyyyyyyyyy # found in twilio.com/console +export TWILIO_PHONE_NUMBER=+17166382453 # replace with your Twilio number +``` + +Note that you must use the `export` command in every command line window +that you want this key to be accessible. The scripts we are writing will +not be able to access the Twilio APIs if you do not have the tokens exported +in the environment where you are running the script. + +There is one more environment variable to set before we can run `app.py`. +We need to use Ngrok as a localhost tunnel so that Twilio's webhook can +send an HTTP POST request to our Django application running on +our local development environment. + +Run Ngrok in a new terminal window, because you will need to keep it +running while we run our other Python code: + +```bash +./ngrok http 8000 +``` + +Ngrok creating a localhost tunnel. + +Copy the HTTPS version of the "Forwarding" URL and set the `BASE_URL` +environment variable value to it. For example, in this screenshot you +would set `BASE_URL` to `https://7764c1810ad3.ngrok.io` using the +following command: + +```bash +export BASE_URL=https://7764c1810ad3.ngrok.io # use your ngrok URL, or domain. no trailing slash +``` + +We also need to update `djtranscribe/djtranscribe/settings.py`'s +`ALLOWED_HOSTS` list to include the Ngrok Forwarding URL otherwise +the [webhook](/webhooks.html) from Twilio asking for instructions +on how to handle the phone call will fail. Open the `settings.py` +file and update the `ALLOWED_HOSTS` with your Ngrok Forwarding +hostname list the following: + +``` +# SECURITY WARNING: keep the secret key used in production secret! +SECRET_KEY = os.getenv('SECRET_KEY', 'development key') + +# SECURITY WARNING: don't run with debug turned on in production! +DEBUG = True + +~~ALLOWED_HOSTS = ['7764c1810ad3.ngrok.io','127.0.0.1','localhost'] + + +# Application definition + +INSTALLED_APPS = [ + 'django.contrib.admin', + 'django.contrib.auth', + 'django.contrib.contenttypes', + 'django.contrib.sessions', + 'django.contrib.messages', + 'django.contrib.staticfiles', + 'caller', +] +``` + +Okay, we can finally re-run our Django web app. Ensure Ngrok is still +running in a different window, your virtualenv is active and that in this +terminal you have your four environment variables set, then run the +`runserver` command in the root project directory where `manage.py` +is located: + +```bash +python manage.py runserver +``` + +Let's make our phone ring by testing the application. + + +## Testing Twilio Programmable Voice Recording +We can test our application by going to localhost on port 8000. +Go to this URL in your web browser, replacing the "14155551234" +with the phone number you want to call, where the person on the +line will be recorded: http://localhost:8000/dial/14155551234. + +That number should now receive a phone call from your Twilio +number. Pick up, record a message that you want to use to test +the transcription, and then hang up. + +If you get an error, make sure all of your environment variables +are set. You can check the values by using the echo command like +this: + +```bash +echo $BASE_URL +``` + +When the call is over, copy the call SID show on the web page +so that we can use it to look up where the recording audio +file is stored. + +Twilio call SID served through the Django web app. + +Go to "localhost:8000/get-recording-url/" with the call SID +at the end. For example, +"localhost:8000/get-recording-url/CAda3f2f49ff4e8ef2be6b726edb998c92". + +Twilio call recording URL. + +Copy the entire output except for the ".json" at the end, then paste +it into the web browser's URL bar, prepended with "api.twilio.com". +For example, +"https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300". +This will bring up the recording. Copy the entire URL and we will use it +as input into the AssemblyAI service. + + +## Transcribing with the AssemblyAI API +We can now use the AssemblyAI API for speech-to-text transcription on +the call recording that was just made. + +[Sign up for an AssemblyAI account](https://app.assemblyai.com/login/) +and log in to the +[AssemblyAI dashboard](https://app.assemblyai.com/dashboard/), then +copy "Your API token" as shown in this screenshot: + +AssemblyAI dashboard. + +We need to export our AssemblyAI API key as an environment variable +so that our Python application can use it to authenticate with their +API. We also need to pass the publicly-accessible URL for the recording, +so we'll set that as an environment variable as well. + +```bash +# make sure to replace this URL with the one for your recording +export ASSEMBLYAI_KEY=your-api-key-here +export RECORDING_URL=https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300 +``` + +Create a new file named `transcribe.py` and write the following code in it: + +```python +import os +import requests + +endpoint = "https://api.assemblyai.com/v2/transcript" + +json = { + "audio_url": os.getenv("RECORDING_URL") +} + +headers = { + "authorization": os.getenv("ASSEMBLYAI_KEY"), + "content-type": "application/json" +} + +response = requests.post(endpoint, json=json, headers=headers) + +print(response.json()) +``` + +The above code calls the AssemblyAI transcription service using +the secret key and passes it the URL with the file recording. +The script prints out the JSON response from the service, +which will contain a transcription ID that we'll use to access +the results after they finish processing. + +Run the script using the `python` command: + +```bash +python transcribe.py +``` + +You will get back some JSON as output, similar what you see here: + +```bash +{'audio_end_at': None, 'acoustic_model': 'assemblyai_default', 'text': None, 'audio_url': 'https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300', 'speed_boost': False, 'language_model': 'assemblyai_default', 'redact_pii': False, 'confidence': None, 'webhook_status_code': None, 'id': 'zibe9vwmx-82ce-476c-85a7-e82c09c67daf', 'status': 'queued', +'boost_param': None, 'words': None, 'format_text': True, 'webhook_url': None, 'punctuate': True, 'utterances': None, 'audio_duration': None, 'auto_highlights': False, 'word_boost': [], 'dual_channel': None, 'audio_start_from': None} +``` + +Find the value contained with the `id` field of the JSON response. We need +that value to look up the final result of our transcription. Copy the +transcription ID and set it as an environment variable to use as input by +the final script: + +``` +# replace with what's found within `id` from the JSON response +export TRANSCRIPTION_ID=aksd19vwmx-82ce-476c-85a7-e82c09c67daf +``` + +We just need a little more Python that looks up the result and we'll be all +done. + + +## Retrieve the AssemblyAI Transcription +AssemblyAI will be busy transcribing the recording. Depending on the size of +the file it can take anywhere from a few seconds to a few minutes for the +job to complete. We can use the following code to see if the job is still +in progress or it has completed. If the transcription is done it will print +the results to the terminal. + +Create a new file named `print_transcription.py` with the following code: + +```python +import os +import requests + +endpoint = "https://api.assemblyai.com/v2/transcript/{}".format(os.getenv("TRANSCRIPTION_ID")) + +headers = { + "authorization": os.getenv("ASSEMBLYAI_KEY"), +} + +response = requests.get(endpoint, headers=headers) + +print(response.json()) +print("\n\n") +print(response.json()['text']) +``` + +The code above in `print_transcription.py` is very similar to the code +in the previous `transcribe.py` source file. imports `os` (operating system) +from the Python standard library, as we did in the previous two files, +to obtain the `TRANSCRIPTION_ID` and `ASSEMBLYAI_KEY` environment variable +values. + +The `endpoint` is the AssemblyAI API endpoint for retrieving +transcriptions. We set the appropriate `authorization` header and +make the API call using the `requests.get` function. We then print +out the JSON response as well as just the text that was transcribed. + +Time to test out this third file. Execute the following command in +the terminal: + +```bash +python print_transcription.py +``` + +Your output will be different based on your recording but you should see a +result in the terminal similar to the following: + +```bash +{'audio_end_at': None, 'acoustic_model': 'assemblyai_default', 'auto_highlights_result': None, 'text': 'An object relational mapper is a code library that automates the transfer of data stored in a relational database tables into objects that are more commonly used in application. Code or MS provide a high level abstraction upon a relational database that allows the developer to write Python code. Instead of sequel to create read update and delete data and schemas in their database developers can use the programming language that they are comfortable with comfortable to work with the database instead of writing sequel statements or short procedures.', 'audio_url': 'https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300', 'speed_boost': False, 'language_model': 'assemblyai_default', 'id': 'zibe9vwmx-82ce-476c-85a7-e82c09c67daf', 'confidence': 0.931797752808989, 'webhook_status_code': None, 'status': 'completed', 'boost_param': None, 'redact_pii': False, 'words': [{'text': 'An', 'confidence': 1.0, 'end': 90, 'start': 0}, {'text': 'object', 'confidence': 0.94, 'end': 570, 'start': 210}, {'text': 'relational', 'confidence': 0.89, 'end': 1080, 'start': 510}, {'text': 'mapper', 'confidence': 0.97, 'end': 1380, 'start': 1020}, {'text': 'is', 'confidence': 0.88, 'end': 1560, 'start': 1350}, {'text': 'a', 'confidence': 0.99, 'end': 1620, 'start': 1500}, {'text': 'code', 'confidence': 0.93, 'end': 1920, 'start': 1620}, {'text': 'library', 'confidence': 0.94, 'end': 2250, 'start': 1860}, {'text': 'that', 'confidence': 0.99, 'end': 2490, 'start': 2220}, {'text': 'automates', 'confidence': 0.93, 'end': 2940, 'start': 2430}, {'text': 'the', 'confidence': 0.95, 'end': 3150, 'start': 2910}, {'text': 'transfer', 'confidence': 0.98, 'end': 3510, 'start': 3090}, {'text': 'of', 'confidence': +0.99, 'end': 3660, 'start': 3480}, {'text': 'data', 'confidence': 0.84, 'end': 3960, 'start': 3630}, {'text': 'stored', 'confidence': 0.89, 'end': 4350, 'start': 3900}, {'text': 'in', 'confidence': 0.98, 'end': 4500, 'start': 4290}, {'text': 'a', 'confidence': 0.85, 'end': 4560, 'start': 4440}, {'text': 'relational', 'confidence': 0.87, 'end': 5580, 'start': 4500}, {'text': 'database', 'confidence': 0.92, 'end': +6030, 'start': 5520}, {'text': 'tables', 'confidence': 0.93, 'end': 6330, 'start': 5970}, {'text': 'into', 'confidence': 0.92, 'end': 7130, 'start': 6560}, {'text': 'objects', 'confidence': 0.96, 'end': 7490, 'start': 7100}, {'text': 'that', 'confidence': 0.97, 'end': 7700, 'start': 7430}, {'text': 'are', 'confidence': 0.9, 'end': 7850, 'start': 7640}, {'text': 'more', 'confidence': 0.97, 'end': 8030, 'start': 7790}, {'text': 'commonly', 'confidence': 0.92, 'end': 8480, 'start': 7970}, {'text': 'used', 'confidence': 0.86, 'end': 8750, 'start': 8420}, {'text': 'in', 'confidence': 0.94, 'end': 9050, 'start': 8840}, {'text': 'application.', 'confidence': 0.98, 'end': 9860, 'start': 9110}, {'text': 'Code', 'confidence': 0.93, 'end': 10040, 'start': 9830}, {'text': 'or', 'confidence': 1.0, 'end': 11210, 'start': 10220}, {'text': 'MS', 'confidence': 0.83, 'end': 11480, 'start': 11180}, {'text': 'provide', 'confidence': 0.94, 'end': 11870, 'start': 11510}, {'text': 'a', 'confidence': 1.0, 'end': 11960, 'start': 11840}, {'text': 'high', 'confidence': 1.0, 'end': 12200, 'start': 11930}, {'text': 'level', 'confidence': 0.94, 'end': 12440, 'start': 12170}, {'text': 'abstraction', 'confidence': 0.95, 'end': 12980, 'start': 12410}, {'text': +'upon', 'confidence': 0.94, 'end': 13220, 'start': 12950}, {'text': 'a', 'confidence': 1.0, 'end': 13280, 'start': 13160}, {'text': 'relational', 'confidence': 0.94, 'end': 13820, 'start': 13280}, {'text': 'database', 'confidence': 0.95, 'end': 14210, 'start': 13790}, {'text': 'that', 'confidence': 0.96, 'end': 14420, 'start': 14150}, {'text': 'allows', 'confidence': 0.99, 'end': 14720, 'start': 14360}, {'text': +'the', 'confidence': 0.56, 'end': 14870, 'start': 14690}, {'text': 'developer', 'confidence': 0.98, 'end': 15290, 'start': 14810}, {'text': 'to', 'confidence': 0.94, 'end': 15410, 'start': 15230}, {'text': 'write', 'confidence': 0.96, 'end': 15680, 'start': 15380}, {'text': 'Python', 'confidence': 0.94, 'end': 16070, 'start': 15620}, {'text': 'code.', 'confidence': 0.98, 'end': 16310, 'start': 16070}, {'text': 'Instead', 'confidence': 0.97, 'end': 17160, 'start': 16500}, {'text': 'of', 'confidence': 0.93, 'end': 17340, 'start': 17130}, {'text': 'sequel', 'confidence': 0.86, 'end': 17820, 'start': 17280}, {'text': 'to', 'confidence': 0.91, 'end': 18090, 'start': 17880}, {'text': 'create', 'confidence': 0.89, 'end': 18450, 'start': 18090}, {'text': 'read', 'confidence': 0.88, 'end': 18840, 'start': 18480}, {'text': 'update', 'confidence': 0.92, 'end': 19290, 'start': 18870}, {'text': 'and', 'confidence': 0.94, 'end': 19590, 'start': 19230}, {'text': 'delete', 'confidence': 0.89, 'end': 19920, 'start': 19530}, {'text': 'data', +'confidence': 0.95, 'end': 20190, 'start': 19890}, {'text': 'and', 'confidence': 0.92, 'end': 20490, 'start': 20250}, {'text': 'schemas', 'confidence': 0.86, 'end': 21000, 'start': 20430}, {'text': 'in', 'confidence': 0.94, 'end': 21210, 'start': 21000}, {'text': 'their', 'confidence': 0.98, 'end': 21510, 'start': 21150}, {'text': 'database', 'confidence': 0.97, 'end': 21900, 'start': 21450}, {'text': 'developers', 'confidence': 0.83, 'end': 23200, 'start': 22420}, {'text': 'can', 'confidence': 0.95, 'end': 23440, 'start': 23200}, {'text': 'use', 'confidence': 0.97, 'end': 23650, 'start': 23410}, {'text': 'the', 'confidence': 0.99, 'end': 23890, 'start': 23590}, {'text': 'programming', 'confidence': 0.97, 'end': 24370, 'start': 23830}, {'text': 'language', 'confidence': 1.0, 'end': 24700, 'start': 24310}, {'text': 'that', 'confidence': 1.0, 'end': 24880, 'start': 24640}, {'text': 'they', 'confidence': 0.99, 'end': 25060, 'start': 24820}, {'text': 'are', 'confidence': 0.85, 'end': 25210, 'start': 25000}, {'text': 'comfortable', 'confidence': 0.92, 'end': 25780, 'start': 25180}, {'text': 'with', 'confidence': 1.0, 'end': 25960, 'start': 25720}, {'text': 'comfortable', 'confidence': 0.94, 'end': 29090, 'start': 28090}, {'text': 'to', 'confidence': 0.84, 'end': 29840, 'start': 29180}, {'text': 'work', 'confidence': 0.95, 'end': 30050, 'start': 29780}, {'text': 'with', 'confidence': 0.98, 'end': 30290, 'start': 30020}, {'text': 'the', 'confidence': 0.69, 'end': 30440, 'start': 30230}, {'text': 'database', 'confidence': 0.98, 'end': 30860, 'start': 30380}, {'text': 'instead', 'confidence': 1.0, 'end': 32780, 'start': 31780}, {'text': 'of', 'confidence': 0.98, 'end': 32900, 'start': 32720}, {'text': 'writing', 'confidence': 0.87, 'end': 33320, 'start': 32870}, {'text': 'sequel', 'confidence': 0.88, 'end': 33860, 'start': 33290}, {'text': 'statements', 'confidence': 0.95, 'end': 34310, 'start': 33800}, {'text': 'or', 'confidence': 0.9, 'end': 34460, 'start': 34250}, {'text': 'short', 'confidence': 0.9, 'end': 34790, 'start': 34430}, {'text': 'procedures.', 'confidence': 0.98, 'end': 35270, 'start': 34760}], 'format_text': True, 'webhook_url': None, 'punctuate': True, 'utterances': None, 'audio_duration': 36.288, 'auto_highlights': False, 'word_boost': [], +'dual_channel': None, 'audio_start_from': None} + + +An object relational mapper is a code library that automates the transfer of data stored in a relational database tables into objects that are more commonly used in application. Code or MS provide a high level abstraction upon a relational database that allows the developer to write Python code. Instead of sequel to create read update and delete data and schemas in their database developers can use the programming language that they are comfortable with comfortable to work with the database instead of writing sequel statements or short procedures. +``` + +That's a lot of output. The first part contains the results of the +transcription and the confidence in the accuracy of each word transcribed. +The second part is just the plain text output from the transcription. + +You can take this now take this base code and add it to any application +that needs high quality text-to-speech transcription. If the results +aren't quite good enough for you yet, check out this tutorial on +[boosting accuracy for keywords or phrases](https://docs.assemblyai.com/guides/boosting-accuracy-for-keywords-or-phrases). + + +## Additional resources +We just finished building a highly accurate transcription application for recordings. + +Next, try out some of these other related [Django](/django.html) tutorials: + +* [Using Sentry to Handle Python Exceptions in Django Projects](/blog/sentry-handle-exceptions-django-projects.html) +* [Tracking Daily User Data in Django with django-user-visit](/blog/track-daily-user-data-django-user-visit.html) +* [How to Quickly Use Bootstrap 4 in a Django Template with a CDN](/blog/bootstrap-4-django-template.html) + +Questions? Let me know via +[a GitHub issue ticket on the Full Stack Python repository](https://github.com/mattmakai/fullstackpython.com/issues), +on Twitter +[@fullstackpython](https://twitter.com/fullstackpython) +or [@mattmakai](https://twitter.com/mattmakai). +See something wrong with this post? Fork +[this page's source on GitHub](https://github.com/mattmakai/fullstackpython.com/blob/master/content/posts/210105-django-accurate-twilio-voice-transcriptions.markdown) +and submit a pull request. + diff --git a/content/posts/210422-monitor-python-aws-lambda-sentry.markdown b/content/posts/210422-monitor-python-aws-lambda-sentry.markdown new file mode 100644 index 000000000..00d3e4cba --- /dev/null +++ b/content/posts/210422-monitor-python-aws-lambda-sentry.markdown @@ -0,0 +1,333 @@ +title: How to Monitor Python Functions on AWS Lambda with Sentry +slug: monitor-python-functions-aws-lambda-sentry +meta: Learn how to monitor your Python 3 functions on AWS Lambda using Sentry. +category: post +date: 2021-04-22 +modified: 2021-04-23 +newsletter: False +headerimage: /img/headers/python-lambda-sentry.jpg +headeralt: The Python, AWS Lambda and Sentry logos are copyright their respective owners. + + +[Amazon Web Services (AWS) Lambda](/aws-lambda.html) is a usage-based +compute service that can run [Python 3](/why-use-python.html) code. Errors +can happen in any environment you are running your application in, so +it is necessary to have reliable [monitoring](/monitoring.html) in place +to have visibility when a problem occurs. + +In this post we will install and configure +[Sentry](https://sentry.io/welcome/)'s application monitoring +service that works specifically for code running on AWS Lambda. + + +## Application Dependencies +A local [development environment](/development-environments.html) is not +required to follow this tutorial because all of the coding and configuration +can happen in a web browser through the +[AWS Console](https://console.aws.amazon.com/console/). + +The example code can be copy and pasted from this blog post or you +can access it on GitHub under the +[Full Stack Python blog-post-examples](https://github.com/fullstackpython/blog-code-examples) +repository within the +[monitor-python-aws-lambda-sentry directory](https://github.com/fullstackpython/blog-code-examples/tree/master/monitor-python-aws-lambda-sentry). + + +## Accessing the AWS Lambda Service +[Sign into your existing AWS account](https://aws.amazon.com/console) +or sign up for a [new account](https://aws.amazon.com/). Lambda +gives you the first 1 million requests for free so that you can execute +basic applications without no or low cost. + +The AWS Lambda landing page. + +When you log into your account, use the search box to enter +"lambda" and select "Lambda" when it appears to get to the right +page. + +Use the search bar to find AWS Lambda. + +If you have already used Lambda before, you will see your existing Lambda +functions in a searchable table. We're going to create a new function so +click the "Create function" button. + +Click the create function button. + +The create function page will give you several options for starting a new +Lambda function. + +The create function details page. + +Click the "Browse Serverless App Repository" selection box, then choose +the "hello-world-python3" starter app from within the +"Public applications" section. + +The create function details page. + +The hello-world-python3 starter app details page should look something +like the following screen: + +Hello world Python3 example app and Lambda function. + +Fill in some example text such as "test" under `IdentityNameParameter` +and click the "Deploy" button: + +Click the deploy button to use the starter app. + +The function will now be deployed. As soon as it is ready we can +customize it and test it out before adding Sentry to capture any errors +that occur during execution. + + +## Testing the starter Python app +Go back to the Lambda functions main page and select your new deployed +starter app from the list. + +List of AWS Lambda functions you have created. + +Find the orange "Test" button with a down arrow next to it like you +see in the image below, and then click the down arrow. Select +"Configure Test Event". + +Configure the test event. + +Fill in the Event name as "FirstTest" or something similar, then +press the "Create" button at the bottom of the modal window. + +Click the "Test" button and it will run the Lambda function with +the parameters from that new test event. You should see something +like the following output: + +```python +Response +"value1" + +Function Logs +START RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914 Version: $LATEST +value1 = value1 +value2 = value2 +value3 = value3 +END RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914 +REPORT RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914 Duration: 0.30 ms Billed Duration: 1 ms Memory Size: 128 MB Max Memory Used: 43 MB Init Duration: 1.34 ms + +Request ID +62fa2f25-669c-47b7-b4e7-47353b0bd914 +``` + +That means the test case was successful, but what happens even if there +is a straightforward mistake in the code, such as trying to access an +undeclared variable? + +Go into the code editor and you should see the starter code like this: + +Code editor within AWS Lambda. + +Update the code with the new highlighted line, which tries to access +a fourth variable, which does not exist in the test configuration +we try to run it with. + +```python +import json + +print('Loading function') + + +def lambda_handler(event, context): + #print("Received event: " + json.dumps(event, indent=2)) + print("value1 = " + event['key1']) + print("value2 = " + event['key2']) + print("value3 = " + event['key3']) +~~ print("value4 = " + event['key4']) + return event['key1'] # Echo back the first key value + #raise Exception('Something went wrong') +``` + +After adding that one new line of code, hit the "Deploy" button, +then the "Test" button. You should see some error output: + +``` +Response +{ + "errorMessage": "'key4'", + "errorType": "KeyError", + "stackTrace": [ + [ + "/var/task/lambda_function.py", + 11, + "lambda_handler", + "print(\"value4 = \" + event['key4'])" + ] + ] +} + +Function Logs +START RequestId: a4e956bd-cce4-403e-b5e7-e95bc3ffa2cb Version: $LATEST +value1 = value1 +value2 = value2 +value3 = value3 +'key4': KeyError +Traceback (most recent call last): + File "/var/task/lambda_function.py", line 11, in lambda_handler + print("value4 = " + event['key4']) +KeyError: 'key4' + +END RequestId: a4e956bd-cce4-403e-b5e7-e95bc3ffa2cb +REPORT RequestId: a4e956bd-cce4-403e-b5e7-e95bc3ffa2cb Duration: 0.81 ms Billed Duration: 1 ms Memory Size: 128 MB Max Memory Used: 43 MB Init Duration: 1.61 ms + +Request ID +a4e956bd-cce4-403e-b5e7-e95bc3ffa2cb +``` + +It is obvious when we are working in the Console that an error just +occurred. However, in most cases an error will happen sporadically +which is why we need a monitoring system in place to catch and report +on those exceptions. + + +## AWS Lambda function monitoring with Sentry +The easiest way to add Sentry to Lambda for this application +is to configure an +[AWS Lambda Layer](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html) +with the necessary dependency for Sentry. Sentry has concise +[documentation on addin gvia Lambda Layers](https://docs.sentry.io/platforms/python/guides/aws-lambda/layer/) +so we will walk through that way to configure it and test it +out. + +First, scroll down to the "Layers" section while in your Lambda +function configuration. Click the "Add a layer" button": + +Add Lambda layer. + +In the "Add layer" screen, select the "Specify an ARN" option. + +Select Specify ARN in the Add Layer screen. + +Now to specify the Amazon Resource Name (ARN), we need to use +the Sentry documentation to get the right configuration string. + +US-East-1 is the oldest and most commonly-used region so I'll +use that here in this tutorial but you should check which one +you are in if you are not certain. + +Select the AWS for the ARN string. + +Copy that value into the Lambda Layer configuration, like this: + +Select the AWS for the ARN string. + +Then press the "Add" button. Now you have the Sentry dependency +in your environment so code that relies upon that library can be +used in the Lambda function. + +Next we need to go into the Sentry dashboard to create a project, +get our unique identifer, and connect it to our Lambda function. + +Sentry can be [self-hosted](https://github.com/getsentry/onpremise) or +used as a cloud service through [Sentry.io](https://sentry.io). We will +use the cloud hosted version because it is quicker than +setting up your own server as well as free for smaller projects. + +Go to [Sentry.io's homepage](https://sentry.io). + +Sentry.io homepage where you can sign up for a free account. + +Sign into your account or sign up for a new free account. You will be at +the main account dashboard after logging in or completing the Sentry sign +up process. + +There are no errors logged on our account dashboard yet, which is as +expected because we have not yet connected our account to our Lambda +function. + +Click "Projects" on the left navigation bar, then "Create Project" +in the top right corner. + +Under "Choose a Platform", select "Serverless" and then "AWS Lambda (Python)" +as shown below: + +Choose AWS Lambda (Python) under the platform options. + +Decide under what criteria it should send error information out of +Lambda. For this tutorial, we will have it send every exception. +Then click the "Create Project." button. + +You can have Sentry handle the instrumentation automatically but +we will handle it manually for our function. On the next screen, Sentry +will provide you with your unique DSN string, which we will need for +our function. + +Copy the Sentry DSN string so we can export it as an environment variable. + +Typically you will want to +[use environment variables on AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html) +to store and access values like your Sentry key. + +Copy the contents of the Sentry DSN string, and go into the Lambda console +to create a new environment variable. To do that, click the "Configuration" +tab within Lambda like you see here: + +Click the Lambda Configuration tab. + +Then click "Edit" and add a new environment variable with the key of `SENTRY_DSN` +and the value of the DSN string that you copied from the Sentry screen. + +Add the environment variable in AWS Lambda. + +Click the "Save" button and go back to your Lambda function code. + +Update your Lambda function with the following highlighted new lines of code +to send errors to Sentry. + +```python +import json +~~import os +~~import sentry_sdk +~~from sentry_sdk.integrations.aws_lambda import AwsLambdaIntegration + +~~SENTRY_DSN = os.environ.get('SENTRY_DSN') +~~sentry_sdk.init( +~~ dsn=SENTRY_DSN, +~~ integrations=[AwsLambdaIntegration()] +~~) + +print('Loading function') + + +def lambda_handler(event, context): + #print("Received event: " + json.dumps(event, indent=2)) + print("value1 = " + event['key1']) + print("value2 = " + event['key2']) + print("value3 = " + event['key3']) + print("value4 = " + event['key4']) + return event['key1'] # Echo back the first key value + #raise Exception('Something went wrong') +``` + +Click the "Deploy" button and then "Test". The code will throw +an error and when we go back to our Sentry dashboard we will +see it captured and viewable for further inspection. + +AWS Lambda exception in the Sentry dashboard. + +It works! Next you will likely want to tune your exception reporting +criteria to make sure you get alerted to the right number of exceptions +if you do not want to see all of them. + + +## What's Next? +We just wrote and executed a Python 3 function on AWS Lambda then +captured the exception message into the Sentry logs. You can +now continue building out your Python code knowing that when something +goes wrong you will have full visibility on what happened. + +Check out the [AWS Lambda section](/aws-lambda.html) for +more tutorials by other developers. + +Further questions? Contact me on Twitter +[@fullstackpython](https://twitter.com/fullstackpython) +or [@mattmakai](https://twitter.com/mattmakai). I am also on GitHub with +the username [mattmakai](https://github.com/mattmakai). + +Something wrong with this post? Fork +[this page's source on GitHub](https://github.com/mattmakai/fullstackpython.com/blob/master/content/posts/210422-monitor-python-aws-lambda-sentry.markdown) +and submit a pull request. diff --git a/content/posts/210823-performance-monitoring-aws-lambda-sentry.markdown b/content/posts/210823-performance-monitoring-aws-lambda-sentry.markdown new file mode 100644 index 000000000..9ede782ef --- /dev/null +++ b/content/posts/210823-performance-monitoring-aws-lambda-sentry.markdown @@ -0,0 +1,261 @@ +title: Application Performance Monitoring AWS Lambda Functions with Sentry +slug: application-performance-monitoring-aws-lambda-functions-sentry +meta: Learn how to use Sentry Application Performance Monitoring on AWS Lambda. +category: post +date: 2021-08-23 +modified: 2021-08-26 +newsletter: False +headerimage: /img/headers/python-lambda-sentry.jpg +headeralt: The Python, AWS Lambda and Sentry logos are copyright their respective owners. + + +[Amazon Web Services (AWS) Lambda](/aws-lambda.html) is a usage-based +computing infrastructure service that can execute +[Python 3](/why-use-python.html) code. One of the challenges of this +environment is ensuring efficient performance of your Lambda Functions. +Application performance monitoring (APM) is particularly useful in these +situations because you are billed based on how long you use the +resources. + +In this post we will install and configure +[Sentry's APM](https://sentry.io/for/performance/) that works via a +[Lambda layer](https://docs.aws.amazon.com/lambda/latest/dg/invocation-layers.html). +Note that if you are looking for error monitoring rather than performance +monitoring, take a look at +[How to Monitor Python Functions on AWS Lambda with Sentry](/blog/monitor-python-functions-aws-lambda-sentry.html) +rather than following this post. + + +## First steps with AWS Lambda +A local [development environment](/development-environments.html) is not +required to follow this tutorial because all of the coding and configuration +can happen in a web browser through the +[AWS Console](https://console.aws.amazon.com/console/). + +[Sign into your existing AWS account](https://aws.amazon.com/console) +or sign up for a [new account](https://aws.amazon.com/). Lambda +gives you the first 1 million requests for free so that you can execute +basic applications without no or low cost. + +The AWS Lambda landing page. + +When you log into your account, use the search box to enter +"lambda" and select "Lambda" when it appears to get to the right +page. + +Use the search bar to find AWS Lambda. + +If you have already used Lambda before, you will see your existing Lambda +functions in a searchable table. We're going to create a new function so +click the "Create function" button. + +Click the create function button. + +The create function page will give you several options for building a +Lambda function. + +The create function details page. + +Click the "Browse Serverless App Repository" selection box, then choose +the "hello-world-python3" starter app from within the +"Public applications" section. + +The create function details page. + +The hello-world-python3 starter app details page should look something +like the following screen: + +Hello world Python3 example app and Lambda function. + +Fill in some example text such as "test" under `IdentityNameParameter` +and click the "Deploy" button: + +Click the deploy button to use the starter app. + +The function will now be deployed. As soon as it is ready we can +customize it and test it out before adding Sentry to capture any errors +that occur during execution. + +Go back to the Lambda functions main page and select your new deployed +starter app from the list. + +List of AWS Lambda functions you have created. + +Find the orange "Test" button with a down arrow next to it like you +see in the image below, and then click the down arrow. Select +"Configure Test Event". + +Configure the test event. + +Fill in the Event name as "FirstTest" or something similar, then +press the "Create" button at the bottom of the modal window. + +Click the "Test" button and it will run the Lambda function with +the parameters from that new test event. You should see something +like the following output: + +```python +Response +"value1" + +Function Logs +START RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914 Version: $LATEST +value1 = value1 +value2 = value2 +value3 = value3 +END RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914 +REPORT RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914 Duration: 0.30 ms Billed Duration: 1 ms Memory Size: 128 MB Max Memory Used: 43 MB Init Duration: 1.34 ms + +Request ID +62fa2f25-669c-47b7-b4e7-47353b0bd914 +``` + +The code was successfully executed, so let's add Sentry's performance +monitoring and test some code that uses it. + + +## Performance monitoring with Sentry +Go to [Sentry.io's homepage](https://sentry.io). + +Sentry.io homepage where you can sign up for a free account. + +Sign into your account or sign up for a new free account. You will be at +the main account dashboard after logging in or completing the Sentry sign +up process. + +Select "Performance" on the left navigation bar, it will take you to the +performance monitoring page. + +Click the 'performance' button on the left side nav. + +Click "Start Setup" then go back over to AWS Lambda to complete the +steps for adding Sentry's Python layer to your Lambda function. + +The easiest way to add Sentry to Lambda for this application +is to configure an +[AWS Lambda Layer](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html) +with the necessary dependency for Sentry. Sentry has concise +[documentation on adding via Lambda Layers](https://docs.sentry.io/platforms/python/guides/aws-lambda/layer/) +so we will walk through that way to configure it and test it +out. + +Scroll down to the "Layers" section while in your Lambda +function configuration. Click the "Add a layer" button": + +Add Lambda layer. + +In the "Add layer" screen, select the "Specify an ARN" option. + +Select Specify ARN in the Add Layer screen. + +Now to specify the Amazon Resource Name (ARN), we need to use +the Sentry documentation to get the right configuration string. + +US-East-1 is the oldest and most commonly-used region so I'll +use that here in this tutorial but you should check which one +you are in if you are not certain. + +Select the AWS for the ARN string. + +Copy that value into the Lambda Layer configuration, like this: + +Select the AWS for the ARN string. + +Then press the "Add" button. You now have the Sentry dependency +in your environment so code that relies upon that library can be +used in the Lambda function. + + +## Testing performance monitoring +Let's change our Python code in the Lambda function and test out +the APM agent. + +Make sure you are signed into your Sentry account and go to +[this specific AWS Lambda set up guide](https://docs.sentry.io/platforms/python/guides/aws-lambda/). + +You will see a "DSN string" that we need to set as an environment +variable on AWS Lambda to finish our setup. Copy the string that +matches your project as shown on that page in the highlighted green +section: + +Copy the Sentry DSN string so we can export it as an environment variable. + +We will +[use environment variables on AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html) +to store and access values like this Sentry DSN key. + +Go into the Lambda console to create a new environment variable. To do +that, click the "Configuration" tab within Lambda like you see here: + +Click the Lambda Configuration tab. + +Then click "Edit" and add a new environment variable with the key of `SENTRY_DSN` +and the value of the DSN string that you copied from the Sentry screen. + +Add the environment variable in AWS Lambda. + +Click the "Save" button and go back to your Lambda function's code editor. + +Replace the code in your Lambda function with the following code: + +```python +import json +import os +import sentry_sdk +import time +from sentry_sdk.integrations.aws_lambda import AwsLambdaIntegration +from sentry_sdk import start_transaction + +SENTRY_DSN = os.environ.get('SENTRY_DSN') +sentry_sdk.init( + dsn=SENTRY_DSN, + traces_sample_rate=1.0, + integrations=[AwsLambdaIntegration()] +) + +print('Loading function') + + +def lambda_handler(event, context): + calc = 1000 + + # this is custom instrumentation, see docs: https://bit.ly/2WjT3AY + with start_transaction(op="task", name="big calculation"): + for i in range(1, 1000): + calc = calc * i + + print(calc) + return event['key1'] # Echo back the first key value +``` + +The above code imports the Sentry dependencies, and then runs both +[automatic instrumentation](https://docs.sentry.io/platforms/python/guides/aws-lambda/performance/instrumentation/automatic-instrumentation/) +and [custom instrumentation](https://bit.ly/2WjT3AY) on the +code. Click the "Deploy" button and then "Test". The code will +successfully execute and when we go back to our Sentry performance +monitoring dashboard we will see some initial results, like this +following screenshot. + +APM results shown in the Sentry dashboard. + +Looks good, you have both the default and the specified transaction +performance recordings in the dashboard, and you can toggle between +them (or other transactions you record) through the user interface. + + +## What's Next? +We just wrote and executed a Python 3 function on AWS Lambda that +used the basics of Sentry APM to get some initial performance +monitoring data. + +Check out the [AWS Lambda section](/aws-lambda.html) for +more tutorials by other developers. + +Further questions? Contact me on Twitter +[@fullstackpython](https://twitter.com/fullstackpython) +or [@mattmakai](https://twitter.com/mattmakai). I am also on GitHub with +the username [mattmakai](https://github.com/mattmakai). + +Something wrong with this post? Fork +[this page's source on GitHub](https://github.com/mattmakai/fullstackpython.com/blob/master/content/posts/210823-performance-monitoring-aws-lambda-sentry.markdown) +and submit a pull request. diff --git a/static/img/201009-twilio-flask-assemblyai/call-recording-url.png b/static/img/201009-twilio-flask-assemblyai/call-recording-url.png new file mode 100644 index 000000000..e2ee688f3 Binary files /dev/null and b/static/img/201009-twilio-flask-assemblyai/call-recording-url.png differ diff --git a/static/img/201009-twilio-flask-assemblyai/dial-call-sid.png b/static/img/201009-twilio-flask-assemblyai/dial-call-sid.png new file mode 100644 index 000000000..a7b527517 Binary files /dev/null and b/static/img/201009-twilio-flask-assemblyai/dial-call-sid.png differ diff --git a/static/img/201009-twilio-flask-assemblyai/ngrok.jpg b/static/img/201009-twilio-flask-assemblyai/ngrok.jpg new file mode 100644 index 000000000..253ed6bb7 Binary files /dev/null and b/static/img/201009-twilio-flask-assemblyai/ngrok.jpg differ diff --git a/static/img/201009-twilio-flask-assemblyai/twilio-console.png b/static/img/201009-twilio-flask-assemblyai/twilio-console.png new file mode 100644 index 000000000..7f0a3a66a Binary files /dev/null and b/static/img/201009-twilio-flask-assemblyai/twilio-console.png differ diff --git a/static/img/210105-django-assemblyai/assemblyai-dashboard.png b/static/img/210105-django-assemblyai/assemblyai-dashboard.png new file mode 100644 index 000000000..509b1af75 Binary files /dev/null and b/static/img/210105-django-assemblyai/assemblyai-dashboard.png differ diff --git a/static/img/210105-django-assemblyai/call-recording-url.png b/static/img/210105-django-assemblyai/call-recording-url.png new file mode 100644 index 000000000..fdf8d4bc4 Binary files /dev/null and b/static/img/210105-django-assemblyai/call-recording-url.png differ diff --git a/static/img/210105-django-assemblyai/dial-call-sid.png b/static/img/210105-django-assemblyai/dial-call-sid.png new file mode 100644 index 000000000..a79dacfb9 Binary files /dev/null and b/static/img/210105-django-assemblyai/dial-call-sid.png differ diff --git a/static/img/210105-django-assemblyai/ngrok.jpg b/static/img/210105-django-assemblyai/ngrok.jpg new file mode 100644 index 000000000..a56e5f21b Binary files /dev/null and b/static/img/210105-django-assemblyai/ngrok.jpg differ diff --git a/static/img/210105-django-assemblyai/twilio-console.png b/static/img/210105-django-assemblyai/twilio-console.png new file mode 100644 index 000000000..7f0a3a66a Binary files /dev/null and b/static/img/210105-django-assemblyai/twilio-console.png differ diff --git a/static/img/210406-python-sentry-aws-lambda/add-env-var.jpg b/static/img/210406-python-sentry-aws-lambda/add-env-var.jpg new file mode 100644 index 000000000..e1f45463d Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/add-env-var.jpg differ diff --git a/static/img/210406-python-sentry-aws-lambda/add-lambda-layer.png b/static/img/210406-python-sentry-aws-lambda/add-lambda-layer.png new file mode 100644 index 000000000..a5eb7c671 Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/add-lambda-layer.png differ diff --git a/static/img/210406-python-sentry-aws-lambda/add-layer-specify-arn.jpg b/static/img/210406-python-sentry-aws-lambda/add-layer-specify-arn.jpg new file mode 100644 index 000000000..b97942149 Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/add-layer-specify-arn.jpg differ diff --git a/static/img/210406-python-sentry-aws-lambda/arn-region.png b/static/img/210406-python-sentry-aws-lambda/arn-region.png new file mode 100644 index 000000000..c44697a65 Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/arn-region.png differ diff --git a/static/img/210406-python-sentry-aws-lambda/aws-lambda-configuration.jpg b/static/img/210406-python-sentry-aws-lambda/aws-lambda-configuration.jpg new file mode 100644 index 000000000..4f5bb0e05 Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/aws-lambda-configuration.jpg differ diff --git a/static/img/210406-python-sentry-aws-lambda/aws-lambda-landing.jpg b/static/img/210406-python-sentry-aws-lambda/aws-lambda-landing.jpg new file mode 100644 index 000000000..799013c14 Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/aws-lambda-landing.jpg differ diff --git a/static/img/210406-python-sentry-aws-lambda/aws-lambda-python.jpg b/static/img/210406-python-sentry-aws-lambda/aws-lambda-python.jpg new file mode 100644 index 000000000..e3ba26649 Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/aws-lambda-python.jpg differ diff --git a/static/img/210406-python-sentry-aws-lambda/configure-test.jpg b/static/img/210406-python-sentry-aws-lambda/configure-test.jpg new file mode 100644 index 000000000..0896d8e30 Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/configure-test.jpg differ diff --git a/static/img/210406-python-sentry-aws-lambda/create-function-detail.png b/static/img/210406-python-sentry-aws-lambda/create-function-detail.png new file mode 100644 index 000000000..7e4c77748 Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/create-function-detail.png differ diff --git a/static/img/210406-python-sentry-aws-lambda/create-function.png b/static/img/210406-python-sentry-aws-lambda/create-function.png new file mode 100644 index 000000000..33706be60 Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/create-function.png differ diff --git a/static/img/210406-python-sentry-aws-lambda/deploy-starter-app.png b/static/img/210406-python-sentry-aws-lambda/deploy-starter-app.png new file mode 100644 index 000000000..4abdfe380 Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/deploy-starter-app.png differ diff --git a/static/img/210406-python-sentry-aws-lambda/functions-list.jpg b/static/img/210406-python-sentry-aws-lambda/functions-list.jpg new file mode 100644 index 000000000..9229b338e Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/functions-list.jpg differ diff --git a/static/img/210406-python-sentry-aws-lambda/hello-world-python3.png b/static/img/210406-python-sentry-aws-lambda/hello-world-python3.png new file mode 100644 index 000000000..f8a9a81fa Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/hello-world-python3.png differ diff --git a/static/img/210406-python-sentry-aws-lambda/lambda-code-editor.jpg b/static/img/210406-python-sentry-aws-lambda/lambda-code-editor.jpg new file mode 100644 index 000000000..4d46f93c7 Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/lambda-code-editor.jpg differ diff --git a/static/img/210406-python-sentry-aws-lambda/lambda-search-bar.png b/static/img/210406-python-sentry-aws-lambda/lambda-search-bar.png new file mode 100644 index 000000000..8388871f3 Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/lambda-search-bar.png differ diff --git a/static/img/210406-python-sentry-aws-lambda/layer-with-arn.png b/static/img/210406-python-sentry-aws-lambda/layer-with-arn.png new file mode 100644 index 000000000..70033365b Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/layer-with-arn.png differ diff --git a/static/img/210406-python-sentry-aws-lambda/sentry-dsn-string.jpg b/static/img/210406-python-sentry-aws-lambda/sentry-dsn-string.jpg new file mode 100644 index 000000000..69a340fb8 Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/sentry-dsn-string.jpg differ diff --git a/static/img/210406-python-sentry-aws-lambda/sentry-error-dashboard.jpg b/static/img/210406-python-sentry-aws-lambda/sentry-error-dashboard.jpg new file mode 100644 index 000000000..4fd7def93 Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/sentry-error-dashboard.jpg differ diff --git a/static/img/210406-python-sentry-aws-lambda/sentry-homepage.jpg b/static/img/210406-python-sentry-aws-lambda/sentry-homepage.jpg new file mode 100644 index 000000000..0fe92307a Binary files /dev/null and b/static/img/210406-python-sentry-aws-lambda/sentry-homepage.jpg differ diff --git a/static/img/210823-sentry-apm-lambda/arn-region.png b/static/img/210823-sentry-apm-lambda/arn-region.png new file mode 100644 index 000000000..4bca3e021 Binary files /dev/null and b/static/img/210823-sentry-apm-lambda/arn-region.png differ diff --git a/static/img/210823-sentry-apm-lambda/layer-with-arn.png b/static/img/210823-sentry-apm-lambda/layer-with-arn.png new file mode 100644 index 000000000..577cbba74 Binary files /dev/null and b/static/img/210823-sentry-apm-lambda/layer-with-arn.png differ diff --git a/static/img/210823-sentry-apm-lambda/performance-results.jpg b/static/img/210823-sentry-apm-lambda/performance-results.jpg new file mode 100644 index 000000000..0e9d3f96d Binary files /dev/null and b/static/img/210823-sentry-apm-lambda/performance-results.jpg differ diff --git a/static/img/210823-sentry-apm-lambda/performance.jpg b/static/img/210823-sentry-apm-lambda/performance.jpg new file mode 100644 index 000000000..7871121d7 Binary files /dev/null and b/static/img/210823-sentry-apm-lambda/performance.jpg differ diff --git a/static/img/210823-sentry-apm-lambda/sentry-dsn-string.png b/static/img/210823-sentry-apm-lambda/sentry-dsn-string.png new file mode 100644 index 000000000..873e68480 Binary files /dev/null and b/static/img/210823-sentry-apm-lambda/sentry-dsn-string.png differ diff --git a/static/img/fsp-fav.png b/static/img/fsp-fav.png index 28a9f6cdb..3a2f75910 100644 Binary files a/static/img/fsp-fav.png and b/static/img/fsp-fav.png differ diff --git a/static/img/headers/python-lambda-sentry.jpg b/static/img/headers/python-lambda-sentry.jpg new file mode 100644 index 000000000..18f0d40db Binary files /dev/null and b/static/img/headers/python-lambda-sentry.jpg differ diff --git a/static/img/logos/bodywork.jpg b/static/img/logos/bodywork.jpg new file mode 100644 index 000000000..821b77ebb Binary files /dev/null and b/static/img/logos/bodywork.jpg differ diff --git a/static/img/logos/cloudflare.png b/static/img/logos/cloudflare.png new file mode 100644 index 000000000..b6a18d47e Binary files /dev/null and b/static/img/logos/cloudflare.png differ diff --git a/static/img/logos/oracle.jpg b/static/img/logos/oracle.jpg new file mode 100644 index 000000000..6e8fec361 Binary files /dev/null and b/static/img/logos/oracle.jpg differ diff --git a/static/img/visuals/cx-oracle.jpg b/static/img/visuals/cx-oracle.jpg new file mode 100644 index 000000000..878e7d285 Binary files /dev/null and b/static/img/visuals/cx-oracle.jpg differ diff --git a/static/img/visuals/oracle-orm-examples.jpg b/static/img/visuals/oracle-orm-examples.jpg new file mode 100644 index 000000000..adae67764 Binary files /dev/null and b/static/img/visuals/oracle-orm-examples.jpg differ diff --git a/theme/templates/article.html b/theme/templates/article.html index 5d7d00cdc..1b2932f1e 100644 --- a/theme/templates/article.html +++ b/theme/templates/article.html @@ -25,7 +25,7 @@ {% if article %}

-

{{ article.title }}

+

{{ article.title }}

{% if article.modified != article.date %} Post updated by diff --git a/theme/templates/base.html b/theme/templates/base.html index f477ca1c7..93e3ad132 100644 --- a/theme/templates/base.html +++ b/theme/templates/base.html @@ -17,9 +17,8 @@ {% block content %}{% endblock %}
{% block lower_banner %}{% endblock %} -
Matt Makai 2012-2020
+
Matt Makai 2012-2022
{% block bottom_banner %}{% endblock %} - {% block js %}{% endblock %} diff --git a/theme/templates/blog.html b/theme/templates/blog.html index 6a20daefb..ed1afb06d 100644 --- a/theme/templates/blog.html +++ b/theme/templates/blog.html @@ -21,7 +21,7 @@

Blog Tutorials

{% endif %} -

{{ a.title }}

+

{{ a.title }}

{% if a.modified != a.date %} Post updated by diff --git a/theme/templates/blog/accurate-twilio-voice-call-recording-transcriptions-assemblyai.html b/theme/templates/blog/accurate-twilio-voice-call-recording-transcriptions-assemblyai.html new file mode 100644 index 000000000..379b4da4b --- /dev/null +++ b/theme/templates/blog/accurate-twilio-voice-call-recording-transcriptions-assemblyai.html @@ -0,0 +1,6 @@ + +Learning Programming +Why use Python? +The Python Programming Language +Python Basic Data Types Tutorial: Strings +APIs diff --git a/theme/templates/blog/application-performance-monitoring-aws-lambda-functions-sentry.html b/theme/templates/blog/application-performance-monitoring-aws-lambda-functions-sentry.html new file mode 100644 index 000000000..e38e97585 --- /dev/null +++ b/theme/templates/blog/application-performance-monitoring-aws-lambda-functions-sentry.html @@ -0,0 +1,7 @@ + +Learning Programming +Web development +AWS Lambda +Sentry +Sentry homepage {% include "blog/external-link.html" %} +Sentry Python Quickstart docs {% include "blog/external-link.html" %} diff --git a/theme/templates/blog/django-accurate-twilio-voice-transcriptions.html b/theme/templates/blog/django-accurate-twilio-voice-transcriptions.html new file mode 100644 index 000000000..8b9e6fcc6 --- /dev/null +++ b/theme/templates/blog/django-accurate-twilio-voice-transcriptions.html @@ -0,0 +1,5 @@ + +Learning Programming +Web development +Django +APIs diff --git a/theme/templates/blog/monitor-python-functions-aws-lambda-sentry.html b/theme/templates/blog/monitor-python-functions-aws-lambda-sentry.html new file mode 100644 index 000000000..e681592c2 --- /dev/null +++ b/theme/templates/blog/monitor-python-functions-aws-lambda-sentry.html @@ -0,0 +1,6 @@ +Learning Programming +Web development +Monitoring +Sentry +Sentry homepage {% include "blog/external-link.html" %} +Sentry Python Quickstart docs {% include "blog/external-link.html" %} diff --git a/theme/templates/blog/sentry-application-performance-monitor-django.html b/theme/templates/blog/sentry-application-performance-monitor-django.html new file mode 100644 index 000000000..6d581b5ea --- /dev/null +++ b/theme/templates/blog/sentry-application-performance-monitor-django.html @@ -0,0 +1,6 @@ +Learning Programming +Web development +Django +Sentry +Sentry homepage {% include "blog/external-link.html" %} +Sentry Performance Monitoring docs {% include "blog/external-link.html" %} diff --git a/theme/templates/choices/demo-software-developers.html b/theme/templates/choices/demo-software-developers.html new file mode 100644 index 000000000..e69de29bb diff --git a/theme/templates/choices/event-streams.html b/theme/templates/choices/event-streams.html new file mode 100644 index 000000000..f7c07c6e9 --- /dev/null +++ b/theme/templates/choices/event-streams.html @@ -0,0 +1,18 @@ +

What topic do you want to learn next?

+
+
+
+ {% include "choices/buttons/databases.html" %} +
+
+
+
+ {% include "choices/buttons/application-dependencies.html" %} +
+
+
+
+ {% include "choices/buttons/development-environments.html" %} +
+
+
diff --git a/theme/templates/choices/oracle.html b/theme/templates/choices/oracle.html new file mode 100644 index 000000000..2ff0b969a --- /dev/null +++ b/theme/templates/choices/oracle.html @@ -0,0 +1,18 @@ +

What's next to get your app running?

+
+
+
+ {% include "choices/buttons/no-sql-datastore.html" %} +
+
+
+
+ {% include "choices/buttons/cascading-style-sheets.html" %} +
+
+
+
+ {% include "choices/buttons/javascript.html" %} +
+
+
diff --git a/theme/templates/code-examples/django.html b/theme/templates/code-examples/django.html index 7b6c539bf..682db8f15 100644 --- a/theme/templates/code-examples/django.html +++ b/theme/templates/code-examples/django.html @@ -198,6 +198,31 @@

django.template.base VariableNode, token_kwargs

+

django.template.context + Context +

+

django.template.defaultfilters + escape, + filesizeformat, + safe, + slugify, + striptags, + title, + truncatechars +

+

django.template.loader + get_template, + render_to_string, + select_template +

+

django.template.loader_tags + BlockNode, + ExtendsNode, + IncludeNode +

+

django.template.loaders.filesystem + Loader +

django.template.response SimpleTemplateResponse, TemplateResponse diff --git a/theme/templates/css/base.css b/theme/templates/css/base.css index a982788e7..5a7ff8055 100644 --- a/theme/templates/css/base.css +++ b/theme/templates/css/base.css @@ -1 +1 @@ -{% raw %}hr,img{border:0}*,:after,:before{box-sizing:border-box}html{-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%;font-size:18px;background:#fefefe}body{margin:0;font:18px Georgia,serif;line-height:1.4;color:#222;padding:0}img{vertical-align:middle}hr{height:0;box-sizing:content-box;margin:21px 0;border-top:1px solid #eee}h1,h2,h3,h4,h5,h6{font-family:"Helvetica Neue",sans-serif;font-weight:500;line-height:1.1;color:#000}h1,h2,h3{margin:32px 0 6px}h1{font-size:40px}h2{font-size:28px}h3{font-size:22px}h4,h5,h6{margin:11px 0;font-size:18px}p{margin:0 0 12px}ol,ul{margin:0 0 10px}code,pre{font:"Courier New",monospace;border-radius:4px;background-color:#f4f9ff;font-size:12px}code{padding:2px 4px;white-space:nowrap}pre{overflow:scroll;white-space:pre;display:block;padding:10px;margin:0 0 11px;line-height:1.4;word-break:break-all;word-wrap:break-word;border:1px solid #ccc}.cn{padding:0 15px 0 15px;margin-right:auto;margin-left:auto}.cn:before,.cn:after{display:table;content:" "}.cn:after{clear:both}.row:before,.row:after{display:table;content:" "}.row:after{clear:both}.c2,.c3,.c4,.c5,.c6,.c7,.c8,.c9,.c10,.c11,.c12{position:relative;min-height:1px;padding:0 15px 0 15px}a{background:transparent;text-decoration:none;border-bottom:1px dotted;color:#444}a:hover{text-decoration:none;color:#000}.ft{padding:0 0 24px;float:right}.sns{font-family:"Helvetica Neue",sans-serif}.sps{font-size:14px}.hd{margin:20px 0 15px 0}.hd>a{border-bottom:none}img.hdr{vertical-align:middle;border:none;height:52px;width:52px;padding:1px}.hdt a,.hdt a:hover{font:72px "Helvetica Neue",sans-serif;font-weight:normal;letter-spacing:.03em;vertical-align:middle;margin-left:5px;color:#000;text-decoration:none;border-bottom:none;line-height:.9em}.bk{margin:0 5px 0 5px}img.nob{border:none}p.banner{font-weight:500;line-height:1.1;color:#fff;font-size:22px;margin:14px 0 18px 0}.bp{line-height:1.3em}{% endraw %} +{% raw %}hr,img{border:0}*,:after,:before{box-sizing:border-box}html{-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%;font-size:18px;background:#fefefe}body{margin:0;font:18px Georgia,serif;line-height:1.4;color:#222;padding:0}img{vertical-align:middle}hr{height:0;box-sizing:content-box;margin:21px 0;border-top:1px solid #eee}h1,h2,h3,h4,h5,h6{font-family:"Helvetica Neue",sans-serif;font-weight:500;line-height:1.1;color:#000}h1,h2,h3{margin:30px 0 6px}h1{font-size:40px}h2{font-size:28px}h3{font-size:22px}h4,h5,h6{margin:11px 0;font-size:18px}p{margin:0 0 12px}ol,ul{margin:0 0 10px}code,pre{font:"Courier New",monospace;border-radius:4px;background-color:#f4f9ff;font-size:12px}code{padding:2px 4px;white-space:nowrap}pre{overflow:scroll;white-space:pre;display:block;padding:10px;margin:0 0 11px;line-height:1.4;word-break:break-all;word-wrap:break-word;border:1px solid #ccc}.cn{padding:0 15px 0 15px;margin-right:auto;margin-left:auto}.cn:before,.cn:after{display:table;content:" "}.cn:after{clear:both}.row:before,.row:after{display:table;content:" "}.row:after{clear:both}.c2,.c3,.c4,.c5,.c6,.c7,.c8,.c9,.c10,.c11,.c12{position:relative;min-height:1px;padding:0 15px 0 15px}a{background:transparent;text-decoration:none;border-bottom:1px dotted;color:#444}a:hover{text-decoration:none;color:#000}.ft{padding:0 0 24px;float:right}.sns{font-family:"Helvetica Neue",sans-serif}.sps{font-size:14px}.hd{margin:20px 0 15px 0}.hd>a{border-bottom:none}img.hdr{vertical-align:middle;border:none;height:52px;width:52px;padding:1px}.hdt a,.hdt a:hover{font:72px "Helvetica Neue",sans-serif;font-weight:normal;letter-spacing:.03em;vertical-align:middle;margin-left:-5px;color:#000;text-decoration:none;border-bottom:none;line-height:.9em}.bk{margin:0 5px 0 5px}img.nob{border:none}p.banner{font-weight:500;line-height:1.1;color:#fff;font-size:22px;margin:14px 0 18px 0}.bp{line-height:1.3em}{% endraw %} diff --git a/theme/templates/index-sidebar.html b/theme/templates/index-sidebar.html index 5fcd2647d..ccfb5ad9e 100644 --- a/theme/templates/index-sidebar.html +++ b/theme/templates/index-sidebar.html @@ -1,3 +1,2 @@
- {% include "sponsor/sentry-assemblyai.html" %}
diff --git a/theme/templates/index.html b/theme/templates/index.html index e021e8703..3c62f365b 100644 --- a/theme/templates/index.html +++ b/theme/templates/index.html @@ -5,15 +5,15 @@ {% endblock %} -{% block css %}{% endblock %} +{% block css %}{% endblock %} {% block banner %} {% endblock %} {% block content %}
-
-

Build, Deploy and Operate Python Applications

+
+

Learn to Build, Deploy and Operate Python Applications

You're knee deep in learning Python programming. The syntax is starting to make sense. The first few ahh-ha! moments hit you as you learn to use @@ -44,7 +44,6 @@

Build, Deploy and Operate Python Applications

What do you need to learn first?

- {% include "index-sidebar.html" %}
diff --git a/theme/templates/nav.html b/theme/templates/nav.html index 529a36676..5b91021c4 100644 --- a/theme/templates/nav.html +++ b/theme/templates/nav.html @@ -1 +1 @@ -
Full Stack Python logo Full Stack Python
{% include "subnav.html" %}
+
Full Stack Python
{% include "subnav.html" %}
diff --git a/theme/templates/sponsor.html b/theme/templates/sponsor.html index 6485b9e0a..bc378aee8 100644 --- a/theme/templates/sponsor.html +++ b/theme/templates/sponsor.html @@ -1,5 +1,4 @@ {% if page.sortorder[0:2] == "01" or page.sortorder[0:2] == "02" or page.sortorder[0:2] == "03" or page.sortorder[0:2] == "04" or page.sortorder[0:2] == "05" or page.sortorder[0:2] == "06" or page.sortorder[0:2] == "50" %} -{% include "sponsor/sentry-assemblyai.html" %} {% include "sponsor/carbon.html" %} {% endif %} {% if false %} diff --git a/theme/templates/sponsor/assemblyai.html b/theme/templates/sponsor/assemblyai.html new file mode 100644 index 000000000..431a8c567 --- /dev/null +++ b/theme/templates/sponsor/assemblyai.html @@ -0,0 +1,7 @@ +
+

Sponsored By

+
+ AssemblyAI logo +

The automatic transcription API loved by Python developers.

+
+
diff --git a/theme/templates/subnav.html b/theme/templates/subnav.html index cbf4e08e9..f57d81fd8 100644 --- a/theme/templates/subnav.html +++ b/theme/templates/subnav.html @@ -1,10 +1,8 @@
All topics | Blog | - Supporter's Edition | @fullstackpython | Facebook | - What's new?